Saravanan666/pypi-org huggingface dataset:
Package: cogdata
CogdataInstallpip install cogdata
sudo `which install_unrarlib.sh`Directory Structure.
├── cogdata_task_task1
│   ├── cogdata_config.json (indicating a task path)
│   ├── merged.bin
│   ├── dataset1
│   │   ├── dataset1.bin
│   │   └── meta_info.json
│   └── dataset2
│       ├── dataset2.bin
│       └── meta_info.json
├── dataset1
│   ├── cogdata_info.json (indicating a dataset path)
│   ├── dataset1.json
│   └── dataset1.rar
└── dataset2
    ├── cogdata_info.json
    ├── dataset2.json
    └── dataset2.zipPipelineThe motivation of this project is to provide lightweight APIs for large-scale NN-based data-processing, e.g. ImageTokenization. The abstraction has 3 parts:Dataset: Raw dataset from other organization in various formats, e.g. rar, zip, etc. The information are recorded atcogdata_info.jsonin its split folder.Task: A task is a collection of "configs, results for different datsets, logs, merged results, and evenly split results". The config of a task are recorded incogdata_info.json. The states (processed, hanging/running, unprocessed)of a dataset in this tasks are inmeta_info.json.DataSaver: The format of saved results. The first option is ourBinSaver, which saves plain bytes with fixed length. It can be read or memmap very fast. The config of DataSaver are also with the task incogdata_info.json.Commandscogdata create_dataset  [-h] [--description DESCRIPTION] --data_files DATA_FILES [DATA_FILES ...] --data_format DATA_FORMAT [--text_files TEXT_FILES [TEXT_FILES ...]] [--text_format TEXT_FORMAT] nameAlias:cogdata data ....data_formatis chosen from class names in cogdata.datasets, e.g.StreamingRarDataset. Texts related options are optional for text-image datasets.cogdata create_task [-h] [--description DESCRIPTION] --task_type TASK_TYPE --saver_type SAVER_TYPE [--length_per_sample LENGTH_PER_SAMPLE] [--img_sizes IMG_SIZES [IMG_SIZES ...]] [--txt_len TXT_LEN]
                           [--dtype {int32,int64,float32,uint8,bool}]
                           task_idAlias:cogdata task ....task_typeandsaver_typeis chosen from class names in cogdata, e.g.ImageTextTokenizationTaskorBinarySaver.cogdata process [-h] --task_id TASK_ID [--nproc NPROC] [--dataloader_num_workers DATALOADER_NUM_WORKERS]
                       [--batch_size BATCH_SIZE] [--ratio RATIO]
                       [datasets [datasets ...]]The i-th proc will be binded to the i-th GPU.cogdata merge [-h] --task_id TASK_IDMerge all the processed data.cogdata list [-h] [--task_id TASK_ID]List all the current datasets in this folder.cogdata clean [-h] [--task_id TASK_ID]Clean the unfinished states of the task.Customized TasksAdd--extra_code PATH_TO_CODEaftercogdata(e.g.,cogdata --extra_code ../examples/convert2tar_task.py [task or process]to execute and register your own task before running the command. Seeexamples/for details.TODO List支持多种不同格式文本处理sphinx 注释文档更详细撰写更精细化的参数管理，将tokenization一般化PPT & 视频介绍Merge 视频处理 [Wenyi]Merge Object detection [Zhuoyi]
Package: coge
How I reuse a existing project as a template effectively?Simple yet powerful code generator.
Make use of existing tool as possiable as we can.Small features:template is runable as normal prj. No more %placehold% stuff.respect .gitignore.support github.com repo.plugin in any language only if you can run it in shell.respect Old keyword case.installpip3 install cogeusageput this line in your~/.bash_profile, and change theyour_template_folderto  the folder where you put your templates.export COGE_TMPLS="<your_template_folder>"demo 1cogejsreactoldkey:newkey:newkey0:newkey1@:destname# What coge does:# - copy $COGE_TMPLS/js/react to $PWD/destname# - change all names from oldkey to newkey,  from  COGE_ARG_0 to newkey0 ,from  COGE_ARG_1 to newkey1demo 2cogejsreact# What coge does:# - Just copy $COGE_TMPLS/js/react to $PWD/appadvanced tricks (optional)work with fzfcg () {
	eval `coge -c $@ | fzf --preview= --bind 'enter:execute-silent(pbcopy <<< {})+abort' ` && pbpaste
}call cg from terminal. and paste it.hook scriptyou can put .coge.after.sh in you template folder.It will execute after copy, which is very handy.For Safety reason. Template from network would need to use -s option to enforcing script executing, only if you know the script is safe.Supported timing and language:.coge.after.sh
.coge.before.sh

.coge.after.py
.coge.before.pyEx:
put .coge.after.sh in your source template#!/bin/bash
echo "init your git repo"
git init.coge.jsonSometimes we do not remember what the string we should repalce,
put a.coge.jsonin your template project.
coge would respect this json file to verify all replacements.
It is just a array with two types, map or string.
all elements in the array must be fullfilled when doing the replacements.
If some strings to be replaced are missed in the command line. Coge would automatically prompt up for user to supply.I think the configuration explains for itself.[{"key":"cli","desc":"this is blala"},"projectname"]helpusage: coge [-h] [-a ARG_PREFIX] [-l] [-c] [-r] [-R] [-s] [-d DEPTH] [-v]
            [magic ...]

       make template link : cd x-engine-module-template && coge -r 
             use template : coge x-engine-module-template xxxx:camera @:x-engine-module-camera  
use git template from net : coge https://www.github.com/vitejs/vite \bvite\b:your_vite @:your_vite  
    

positional arguments:
  magic                 newkey:oldkey or @:folder_name

optional arguments:
  -h, --help            show this help message and exit
  -a ARG_PREFIX, --arg_prefix ARG_PREFIX
                        ex: COGE_ARG_
  -l, --list            list folders
  -c, --cmd             cmd
  -r, --link_tplt       link `cwd` to $COGE_TMPLS
  -R, --unlink_tplt     unlink `cwd`
  -s, --script_from_net
                        alllow script from net
  -d DEPTH, --depth DEPTH
                        list depth
  -v, --version         version
Package: cogee
cogee A Simple CLI for COGs in GEEcogeeis a command-line interface (CLI) designed to simplify the process of registering Cloud Optimized Geotiff (COG) assets into Google Earth Engine (GEE) image collections. It streamlines the workflow of ingesting and managing geospatial data within GEE, enabling users to efficiently access, analyze, and visualize COG assets in their GEE projects.cogeesimplifies the workflow of registering COG assets into GEE image collections, making it easier for users to harness the power of Google Earth Engine for geospatial analysis and visualization.You can find information on the tools inthe readme docsNotesThis tools is designed for general purpose registration of COGs in a GCS bucketCurrently there is no way to pass the properties information to the registration pipelineThis will be in active development and features will be added over time.v1.0.0 marks the first major release of this tool.
Package: cogen
This is a library for network oriented, coroutine based programming.cogen’s goal is to enable writing code in a seemingly synchronous and easy
manner in the form of generators that yield calls and receive the result
from that yield. These calls translate to asynchronous and fast os calls
incogen’s internals.Notable featuresa WSGI server, HTTP1.1 compliant, with asynchronous extensionsepoll, kqueue, select, i/o completion ports, sendfile behind the scenesa couple of useful classes for putting the coroutine to sleep, wait for
signals, queues, timeouts etc.Quick introductionA coroutine is just a generator wrapped in a helper class:from cogen.core.coroutines import coroutine

@coroutine
def mycoro(bla):
    result = yield <operation>
    result = yield <operation>theoperationinstructs the scheduler what to do with the coroutine:
suspend it till something happens, add another coro in the scheduler, raise
a event and so on.if aoperationhas a result associated then the yield will return that
result (eg. a string or a (connection, address) tuple) otherwise it will
return the operation instance.Echo server examplefrom cogen.core import sockets
from cogen.core import schedulers
from cogen.core.coroutines import coroutine

@coroutine
def server():
    srv = sockets.Socket()
    print type(srv)
    srv.bind(('localhost',777))
    srv.listen(10)
    while 1:
        print "Listening..."
        conn, addr = yield srv.accept()
        print "Connection from %s:%s" % addr
        m.add(handler, args=(conn, addr))

@coroutine
def handler(sock, addr):
    yield sock.write("WELCOME TO ECHO SERVER !\r\n")

    while 1:
        line = yield sock.readline(8192)
        if line.strip() == 'exit':
            yield sock.write("GOOD BYE")
            sock.close()
            return
        yield sock.write(line)

m = schedulers.Scheduler()
m.add(server)
m.run()Documentationhttp://cogen.googlecode.com/svn/trunk/docs/build/index.htmlDevelopmentTakes place at:http://code.google.com/p/cogen/Grab the latest and greatest fromtrunkwith:easy_install cogen==dev
Package: cogeno
Cogeno, the inline code generation tool, processes Python or Jinja2 ‘snippets’
inlined in your source files. It can also access CMake build
parameters and device tree information to generate source code automatically
tailored and tuned to a specific project configuration.Cogeno can be used, for example, to generate source code that creates
and fills data structures, adapts programming logic, creates
configuration-specific code fragments, and more.
Package: co-geo
co-geoPython library to solve 2d/3d coordinate geometryInstallationpip install co-geoRequirementspipinstallutnamtteWays to importMethod 1importco-geoprint(co-geo.area_3d(...))Method 2fromco-geoimportareaprint(area(...))Method 3fromco-geoimportareaasxyprint(xy(...))Centroid_2d# Centroid_2d(x1, y1, x2, y2, x3, y3)a=co-geo.Centroid_2d(0,2,3,7,6,3)print("x coordinate:",a.x)print("y coordinate:",a.y)Output:34Centroid_3d# Centroid_3d(x1, y1, z1, x2, y2, z2, x3, y3, z3)a=co-geo.Centroid_3d(0,2,11,3,7,4,6,3,6)print("x coordinate:",a.x)print("y coordinate:",a.y)print("z coordinate:",a.z)Output:347Orthocentre_2d# Orthocentre_2d(x1, y1, x2, y2, x3, y3)a=co-geo.Orthocentre_2d(2,5,2,8,14,5)print("x coordinate:",a.x)print("y coordinate:",a.y)Output:25Orthocentre_3d# Orthocentre_3d(x1, y1, z1, x2, y2, z2, x3, y3, z3)a=co-geo.Orthocentre_3d(2,-5,-3,2,7,-3,14,-5,6)print("x coordinate:",a.x)print("y coordinate:",a.y)print("z coordinate:",a.z)Output:2-5-3▶️Package's YouTube tutorial hereChange Log0.0.1 (17/06/2023)First Release
Package: cogeotiff
## Cloud Optimized GeoTiff utility tools## Requirements- click- gdal## Install- pypi: `pip install cogeotiff`- source code: `python setup.py install`## Usage### Python PAI1. create cog fileexample:```pythonfrom cogeotiff.cog import create_cogsrc_path = '~/path_to_image/test.tif'dest_path = '~/path_to_save/test.tiff.cog'create_cog(src_path, dest_path, compress='jpeg')```2. validate cog file```pythonfrom cogeotiff.validate_cogeotiff import validate_cogcog_path = '~/path_to_cog.tif.cog'result = validate_cog(cog_path)```### Command tools1. create cog file```bashUsage: cog create [OPTIONS] SRC_PATH DST_PATHOptions:--overview-resampling TEXT  resampling method for create pyramid default:nearest--overview-level INTEGER    levels to build--nodata INTEGER            Assign a specified nodata value to output bands.default: 0--block-size INTEGER        tiled size, default: 512--compress TEXT             compress method: JPEG,RAW,DEFLATE,LZW,WEBP,ZSTD,PACKBITS,default: raw--help                      Show this message and exit.```2. validate cog file```bashUsage: cog create [OPTIONS] COG_PATH```## Q&A- Q: why not use `GDAL BuildOverviews` or `Rasterio` in script?- A: Because for large file such as 100GB, it's take long time to generate overview and translate to tiled GeoTiff. There is no progress infomataion use `Rasterio` or `GDAL BuildOverviews` methods.- Q: why not use [rio-cogeo](https://github.com/cogeotiff/rio-cogeo)?- A: Cause it use `rasterio MemoryFile()`, when the file is too large, it will cause insufficient memory.
Package: cog-generator
Cloud Optimized GeoTiff GeneratorOverviewThis package wrapped around the native GDAL library enables Cloud Optimized GeoTiffs generation at ease into your Python projects.InstallationYou can install the package via pip:pipinstallcog_generationTo generate a cog:fromcog_generatorimportgenerate_coginput_tif_list=["sample1.tif","sample2.tif"]output_tif_path="output_cog.tif"options={"compress":"lzw","level":9}message,status_code=generate_cog(input_tif_list,output_tif_path,options)Parameters available for optionscompress: Defaults tolzw. Options includenone,lzw,jpeg,deflate,zstd,webp,lerc,lerc_deflate,lerc_zstd,lzma.level: Defaults to6for DEFLATE/LZMA,9for ZSTD. Compression level options depend on the compression type. Lower values result in faster compression but less efficient compression rate. 1 is the fastest.QUALITY: Defaults to75. JPEG/WEBP quality setting. For WEBP,QUALITY=100automatically turns on lossless mode.num_threads: Defaults toall_cpus. Enable multi-threaded compression by specifying the number of worker threads.PREDICTOR: Defaults toNO. Set the predictor for LZW, DEFLATE, and ZSTD compression.bigtiff: Defaults toIF_NEEDED. Control whether the created file is a BigTIFF or a classic TIFF.zoom_level: (GDAL >= 3.5) Zoom level number (starting at 0 for coarsest zoom level).
Package: cog-hf-template
cog-huggingface-templateTools to help deploy Hugging Face models to replicate.RequirementsGPU environmentCogInstallationPyPI Installpip install cog_hf_templateDev InstallClone the repo:git clone https://github.com/replicate/cog-huggingface-template
cd cog-huggingface-template
pip install -e .Run testspip install pytestpytest -vv testsUsageNOTE (Oct 19) - examples are out of date. Will update soon.For now, see theexamples directoryfor a working example.Mirroring Weights from HF to Replicate StorageHere's a quick tip on how to mirror weights from Hugging Face to GCPYou can use a minimal install of this package to avoid the extra dependencies...# required deps for mirroring weights/CLI
pip install --no-cache huggingface-hub==0.18.0 google-cloud-storage==2.12.0 hf-transfer==0.1.3 fire==0.5.0

# Minimal version of the package
pip install --no-deps --no-cache --upgrade cog-hf-templateYou will also need pget installedcurl -o /usr/local/bin/pget -L "https://github.com/replicate/pget/releases/download/v0.0.2/pget" && chmod +x /usr/local/bin/pgetThen you can run the following command to mirror the weights:cog-hf-template mirror-weights \
    openai/whisper-large-v3 \
    your-bucket \
    --gcp-project-name your-project \
    --config_save_path whisper_large_v3_config.jsonSeecog-hf-template mirror-weights --helpfor more details.In the example above, we mirroropenai/whisper-large-v3to the bucketyour-bucketin the projectyour-project. We also save the model config towhisper_large_v3_config.jsonso that we can use it later to load the model (as you'll need to know the files to download).The config looks like this:{"hf_model_id":"openai/whisper-large-v3","task":"automatic-speech-recognition","gcp_bucket_weights":"gs://your-bucket/openai/whisper-large-v3/1940b900b8d4d8fdf72a31be85a0d223b9488e00","trust_remote_code":true,"remote_filenames":[".gitattributes","README.md","added_tokens.json","config.json","flax_model.msgpack","generation_config.json","merges.txt","model.fp32-00001-of-00002.safetensors","model.fp32-00002-of-00002.safetensors","model.safetensors","model.safetensors.index.fp32.json","preprocessor_config.json","pytorch_model.bin","pytorch_model.bin.index.fp32.json","pytorch_model.fp32-00001-of-00002.bin","pytorch_model.fp32-00002-of-00002.bin","special_tokens_map.json","tokenizer_config.json","vocab.json"]}Note the files are accessible through the delivery URLs, and will thegs://your-bucketprefix will be replaced by the appropriate prefix before being downloaded.importjsonfrompathlibimportPathfromcog_hf_template.download_utilsimportmaybe_pget_weightsconfig=json.loads(Path("whisper_large_v3_config.json").read_text())maybe_pget_weights(path="whisper_large_v3",remote_path=config["gcp_bucket_weights"],remote_filenames=config["remote_filenames"],)
Package: coghotswap
CogHotSwapWork in progressYou ever wish that your discord bot cogs would just reload by themselves? Well now you can, by adding just one line of code to your bot we will automate the reloading of all your cogs, and hopefully give you feed back (Coming Soon Feature).This project was built forPycorda fork ofDiscord.pyInstallationgitclonehttps://github.com/TropicalShadow/CogHotSwapcdCogHotSwap
python-mpipinstall-U.or if you have git, add this to your requirements.txtgit+https://github.com/TropicalShadow/CogHotSwapComing Soon FeaturesList of unloaded cogs that can be loaded or have erroredEnchance the reloading of the filesExamplesherefromdiscord.ext.commandsimportBotfromcoghotswapimportWatcherbot=Bot("!")Watcher(bot,path="cogs")bot.run("token")Need Support?Contact me on discordRealName_123#2570
Package: cogment-agent-toolbox
Agent ToolboxOverviewThe agent toolbox generates tensors from the observation and action space. These tensors or vectors can be used directly to Tensorflow or PyTorch.Setuppipinstallcogment_agent_toolboxUse TensorGenerator class in your agentCreate a generatorfromagent_toolbox.tensor_generatorimportTensorGeneratorgenerator=TensorGenerator()Get the action size and observation size of the environment. These functions are uselfull to create the neural net with the right sizes.generator=TensorGenerator()state_size=generator.get_state_size()action_size=generator.get_action_size()Compute state tensor from the environment observation or compute action tensor from an environment actionstate=generator.compute_tensor(obs)Compute action from a tensor, useful function when the type of action  is continuousaction_from_tensor=generator.compute_action_from_tensor(action_tensor)Discrete Actions utilsConsult all possible actions generated for a classification problemgenerator.get_possible_actions()Returns the number of actions generated for a classification problemclassification_size=generator.get_classification_size()equivalent tolen(generator.get_possible_actions())Compute a choice for a classification problem, from an action tensor. The returned value is an index in the possible action list.generator.get_possible_actions()choice=generator.compute_choice_from_action(action_tensor)Convert the selected action index in action required by the environment. The parameterchoiceis an index from 0 to n,
n beinggenerator.get_classification_size().generator.convert_choice_in_action(choice)ExamplesUsing action tensor directly, useful for continuous action type.fromagent_toolbox.tensor_generatorimportTensorGenerator...defdecide(self,observation:cog_settings.actor_classes._otsa.observation_space):# Create generatorgenerator=TensorGenerator()# Transform the observation space in tensorobs_tensor=generator.compute_tensor(observation)print("Observation tensor: ",obs_tensor)# Pick an action from a tensoraction_size=generator.get_action_size()action_tensor=[random.random()foriinrange(action_size)]# Transform the tensor in actionaction_from_tensor=generator.compute_action_from_tensor(action_tensor)print("Action from tensor : ",action_from_tensor)Solving classification problem by using a list generated actions, useful for discrete action type.defdecide(self,observation:cog_settings.actor_classes._otsa.observation_space):# Create generatorgenerator=TensorGenerator()# Transform the observation space in tensorobs_tensor=generator.compute_tensor(observation)print("Observation tensor: ",obs_tensor)# Pick an action from a classification problemclassification_size=generator.get_classification_size()action_choice=random.randrange(classification_size)# Transform the choice in actionaction_from_choice=generator.convert_choice_in_action(action_choice)print("Action from choice : ",action_from_choice)
Package: cog-nba
Cache-or-GET NBA WrapperThis is a simple wrapper over the stats.nba.com API which caches requests made.Visit myPersonal Websitefor other projects.DependenciesThis project usesrequestsandsqlalchemyInstructionspip install cog_nbaAvailable CommandsThe following command scan be used at this stage.cog_nba.nba.get_current_players()cog_nba.nba.get_all_players()cog_nba.nba.get_player_career_data(player_id)Alternatively, you can access any given endpoint by using:cog_nba.nba.get(endpoint, params)
Package: cognet
Info:Draft link will be posted hereAuthor:ZeD@UChicago<zed.uchicago.edu>Description:Application of quasinets (https://pypi.org/project/quasinet/) for analysis of emergent structures in survey responses with application in  adversarial contexts.Documentation:https://zeroknowledgediscovery.github.io/cognet/Usage:from cognet.cognet import cognet as cg
from cognet.dataFormatter import dataFormatter
from cognet.model import model


data = dataFormatter(samples=GSSDATA,
            test_size=0.5)
Package: cognica
Cognica Python ClientA Python client for Cognica database server.Installation$pipinstallcognicaCurrent StatusThe current version is fully functional and stable for the production environment. However, the API is still subject to change and may break backward compatibility in the next release.Current Versionimportcognica# Establish connection to a local database server.channel=cognica.Channel("localhost",10080)# List collections.db=cognica.DocumentDB(channel)collections=db.list_collections()forcollectionincollections:print(collection)# Find a specific document in the Wikipedia collection.collection=cognica.find_collection("Wikipedia")df=collection.find({"page_id":42},to_pandas=True)print(df)Next Version (Work in progress)importcognica# Establish connection to a local database server.channel=cognica.Channel("localhost",10080)# Login to the database.ws=cognica.Workspaces(channel,"user_id","password")# List workspaces.workspaces=ws.list_workspaces()forworkspaceinworkspaces:print(workspace)# Open the workspace and list the collections.workspace=ws.find_workspace("test")db=cognica.DocumentDB(workspace)collections=db.list_collections()forcollectionincollections:print(collection)# Find a specific document in the Wikipedia collection.collection=cognica.find_collection("Wikipedia")df=collection.find({"page_id":42},to_pandas=True)print(df)