Saravanan666/pypi-org huggingface dataset:
Package: batchfilerename
This is a simple batch file renaming utility written in pure Python. You can install with:pip install batchfilenameInstall from github:pip install -e git+https://github.com/dmnfarrell/batchfilename.git#egg=batchfilenameHow to useRun using the commandbatchfilename. A window with two panes will appear. Select the folder where the files are to be renamed. On the left the files will be listed and on the right a preview of the renamed files is shown (without full path for ease of viewing). You can then select the symbols to find and replace with, which will be applied to all files. Filter the files to be renamed if needed,*.*means all files.Always use ‘preview’first to check the results before executing as some file name changes might not be reversible. Though you should be able to reverse the last run using the undo button.Other featuresThe ‘occurences’ option allows you to only replace a specific number of instances of a symbol in the name.Undo the previous renaming step (assuming you have not quit the program)Recursively load a folder
Package: batch-football-data
Batch Football DataPython Data Models for interacting with the Football Stats Database.ModulesThe Football Data library contains two modules:ModelsRepositoriesModels ModuleThe Models Module contains ORM Classes representing the tables within the database mapped for use in SQL Alchemny. The following Data Classes
are available:Players - Represents a player in the league.Position - Lookup for the Player's PositionSchedule - Represents a given game played during the week in a season.Statistic - A Ststistic for either a player or a TeamStatisticCategory - Category of Statistic (Defense, Offense)StatisticCode - Code Value related to the Statistic (Passing Yard, Rushing Yards, etc)Team - Team in the LeagueTypeCode - Defines the Season type for a schedule entry (Preseason, Regular, Post Season)TeamStaff - Establishes a relationship for a Player to a Team for a given YearLeague - Defines a League for teamsTeamLeague - Establishes a linkage between a Team and League.Repositories ModuleThe repositories module contains individual repository class for each of the Model classes for interacting with items in the database.
Each repository contains methods for saving single and multiple items. They also contain methods for validating the item is present or not in the
database.
Package: batch-gcd
Batch GCDOverviewA pure Python implementation of DJB's Batch GCD algorithm.InstallationDownload and install withpip.Or, download from source and install withmake install(standard, user) ormake install_dev(editable, system).UsageThis is a library and cannot be invoked directly.Test withmake test.Thebatch_gcdmodule exposes abatch_gcdfunction which takes integers and returns a list of their GCDs at the corresponding index.>>># Example batch_gcd usage>>>frombatch_gcdimportbatch_gcd>>>batch_gcd(1909,2923,291,205,989,62,451,1943,1079,2419)[1909,1,1,41,23,1,41,1,83,41]This calculation involves two intermediate steps: creating a product tree and creating a remainder tree.
These functions are also exposed, asproductsandremainders.productstake integers and returns a product tree,remainderstakes an integer and a product tree and returns a list of remainders.ResourcesHow to Find Smooth Parts of IntegersFactHacks: Batch GCDFactHacks: RSA Factorization in the Real World
Package: batchglm
Fast and scalable fitting of over-determined generalized-linear models (GLMs)batchglm was developed in the context ofdiffxpyto allow fast model fitting for differential expression analysis for single-cell RNA-seq data. However, one can use batchglm or its concepts in other scenarios where over-determined GLMs are encountered. batchglm is based on TensorFlowInstallationInstalltensorflow, see below. Please use the pip installation if you are unsure.Clone the GitHub repository of batchglm.cd into the clone.pip install -e .Tensorflow installationTensorflow can be installed like any other package or can be compiled from source to allow for optimization of the software to the given hardware. Compiling tensorflow from source can significantly improve the performance, since this allows tensorflow to make use of all available CPU-specific instructions. Hardware optimization takes longer but is only required once during installation and is recommended if batchglm is used often or on large data sets. We summarize a few key steps here, an extensive up-to-date installation guide can be found here:https://www.tensorflow.org/install/Out-of-the-box tensorflow installationYou can installtensorflowvia pip or via conda.pipCPU-only:pip install tensorflowGPU:pip install tensorflow-gpuHardware-optimized tensorflow installation (compiling from source)Please refer tohttps://www.tensorflow.org/install/.
Package: batchhttp
batchhttp provides parallel fetching of HTTP resources through MIME multipart
encoding.This package’sBatchClientapplies standard MIME multipart encoding to HTTP
messages, providing a standards-conservant technique for making parallelizable
HTTP requests over a single proxy connection.To make a batch request, open a new request on aBatchClientinstance and
add your subrequests, along with callbacks that will receive the subresponses.
Once all parallelized requests are added, complete the request; the request is
made and the batched subresponses are provided to your callbacks.This is an implementation of the draft specification for batch HTTP request
processing available at:http://martin.atkins.me.uk/specs/batchhttp
Package: batch-ida
Batch-IDAA python library for generate ida pro files in batch mode & compare executable files use bindiff in batch mode.FeaturesFast generate idb/i64 files from binary files in batch mode (Multiprocess supported)IDA Pro Plugin Script supportUse bindiff to batch compare idb filesRoughly read the comparison resultsRequirementsWindows environment (i don't know if it works on linux)IDA pro 7/8 (Only tested on IDA pro 7.7)BinDiff 7 (Optional, used to compare executable files in batch mode)Installpip install --upgrade batch-idaUsageBI_IDAGenerate .idb/.i64 files (and run plugin) in batch mode.frombatch_idaimportBI_IDA# create a BI_IDA object with ida pro pathbi=BI_IDA('C:\Tools\IDA Pro')# optional# bi.set_script(r'.\example_script.py')   # run IDA script# bi.max_subprocess = 8                   # set max subprocessbi.batch_idb_fromdir('your/binary/files/dir')# generate idb files from binary filesBI_DircmpCompare files in dir_a & dir_b, and move different files to dst_a & dst_b.frombatch_idaimportBI_Dircmpdir_a=r'ver_A.rom/squashfs-root/usr/lib/'dir_b=r'ver_B.rom/squashfs-root/usr/lib/'dst_a=r'compare/ver_A'dst_b=r'compare/ver_B'bid=BI_Dircmp(dir_a,dir_b,dst_a,dst_b)bid.cmp()BI_BindiffGenerate .idb and .bindiff files in batch mode.frombatch_idaimportBI_Bindiffbib=BI_Bindiff()# Set ida pro and bindiff pathbib.set_ida_path('C:\Tools\IDA Pro')bib.set_bindiff_path('C:\Program Files\BinDiff')dst_a=r'compare/ver_A'dst_b=r'compare/ver_B'# dst_a and dst_b are the directories where the binary files to be compared are locatedoutput=bib.batch_bindiff(dst_a,dst_b)# output is a directory containing bindiff files(sqlite3)BI_AnalyzerAnalyze Bindiff files (sqlite3 file format) in batch mode and print the results.frombatch_idaimportBI_Analyzerbia=BI_Analyzer(r'C:\Users\zzhihan\Desktop\test\4_4010_vs_4_4040')# bia.print_base_info()# print the comparison resultsinfo_list=bia.get_info_list()foriininfo_list:ifi['total_func']&i['func_dif']&i['libfunc_dif']:print("%.02f\t%.2f\t%d\t%d\t%d\t%s"%(i['similarity'],i['confidence'],i['total_func'],i['func_dif'],i["libfunc_dif"],i['name']))# print the comparison results that are not 95% similarelifi['similarity']<0.95andi['similarity']!=0.0:print("%.02f\t%.2f\t%d\t%d\t%d\t%s"%(i['similarity'],i['confidence'],i['total_func'],i['func_dif'],i["libfunc_dif"],i['name']))
Package: batchify
batchify: structured data in pytorchPyTorch can already batchify tensors (and tuples of tensors), but what about arbitrary classes? If your neural network is dealing with complex datatypes, structuring your data in classes is the solution. With batchify, you can seemlessly return classes in a pytorch dataset.As an example, let's say you want your neural network do something with people. People, as we all know, are faces and namesimporttorchfrombatchifyimportBatch,BatchableMAX_NAME_LEN=128IMG_SIZE=256classPerson(Batchable):face:torch.Tensorname:torch.Tensordef__init__(self):# not a very interesting personself.face=torch.zeros((3,IMG_SIZE,IMG_SIZE))self.name=torch.zeros((MAX_NAME_LEN,))Now here's the fun part: we can make a batch of people. This automatically batchifies both the face and the namedave=Person()rhonda=Person()batch=Batch([dave,rhonda])print(len(batch))print(dave.name.shape)print(batch.name.shape)# notice the extra batch dimensionprint(batch[0].name.shape)# un-batchification2
torch.Size([128])
torch.Size([2, 128])
torch.Size([128])But what about a custom person dataset? Pretty easy with the batchify dataloaderfrombatchifyimportDataLoaderclassPersonDataset(torch.utils.data.Dataset):def__len__(self):return16def__getitem__(self,index):returnPerson()batch_size=8dataset=PersonDataset()loader=DataLoader(dataset,batch_size=batch_size)forbatchinloader:print(batch.face.shape)torch.Size([8, 3, 256, 256])
torch.Size([8, 3, 256, 256])This is all great if you want to input a Person into your network. But what if you want tooutputa person?(warning: this functionality only works if you have correct type annotations on your Batchable classes)out_batch=Batch(Person,face=torch.zeros((batch_size,3,IMG_SIZE,IMG_SIZE)),name=torch.zeros((batch_size,MAX_NAME_LEN)))print(len(out_batch))print(out_batch[0].name.shape)8
torch.Size([128])
Package: batchimage
Batch Image ProcessingA Python Package for Fast Parallel Processing of Images. It's Simple, easy and extremely fast since it uses, all the cores in your CPU. batchimage uses opencv-python as its main dependency and few python core modules.Python Package Index BadgesGithub BadgesPython Package Index Installpip install batchimageNote :This is a pre-alpha version of the package, the api will change without any notice. Please use this package with caution.Timeline for DevelopmentPresentSeptember 2022October 2022November'22
Package: batch-image-processing
Batch Image ProcessorThis is a Python library that provides methods toresize, crop, convert, rotate, and blurimages in a specified folder and its subfolders using thePillow(PIL Fork) library.RequirementsPython >=3.7typingmoduleosmodulePillow(Pillow (PIL) Fork) >= 9.0.0This package usingPillowlibrary. Hence, it isrequiredto install this package before installing batch image processing package. Please check the following documentation to install Pillow libraryhttps://pillow.readthedocs.io/en/stable/InstallationYou can simply intall this package by usingpippip install batch-image-processingUsageImport the package to your file:from batch_image_processing import ImageProcessorCreate an object ofImageProcessorclass. When you creating the instance, you need to pass two arguments.path/to/image_folderpath/to/output_folderprocessor = ImageProcessor("images/images-to-crop", "images/processed-images")After that, call to the functions according to your preferencesResize Imagesprocessor.resize_images(100, 100)Crop Imagesprocessor.crop_images(50, 50, 150, 150)Convert images to another typeprocessor.convert_images("png")Rotate Imagesprocessor.rotate_images(90)Blur Imagesprocessor.blur_images(5)Grayscale Imagesprocessor.grayscale_images()ContributingContributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make aregreatly appreciated.If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!Fork the ProjectCreate your Feature Branch (git checkout -b feature/AmazingFeature)Commit your Changes (git commit -m 'Add some AmazingFeature')Push to the Branch (git push origin feature/AmazingFeature)Open a Pull RequestAuthorAuthorAkash ManujayaLicensePermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Package: batch-inference
Batch Inference ToolkitBatch Inference Toolkit(batch-inference) is a Python package that batches model input tensors coming from multiple users dynamically, executes the model, un-batches output tensors and then returns them back to each user respectively. This will improve system throughput because of better compute parallelism and better cache locality. The entire process is transparent to developers.When to useWhen you want to host Deep Learning model inference on Cloud servers, especially on GPUWhy to useIt can improve your server throughput up to multiple timesAdvantage of batch-inferencePlatform independent lightweight python libraryOnly few lines code change is needed to onboard using built-inbatching algorithmsFlexible APIs to support customized batching algorithms and input typesSupportmulti-process remote modeto avoid python GIL bottleneckTutorials and benchmarks on popular models:ModelThroughput Comparing to BaselineLinksBert Embedding4.7xTutorialGPT Completion16xTutorialInstallationInstall from Pippython-mpipinstallbatch-inference--upgradeBuild and Install from Source(for developers)gitclonehttps://github.com/microsoft/batch-inference.git
python-mpipinstall-e.[docs,testing]# if you want to format the code before commitpipinstallpre-commit
pre-commitinstall# run unittestspython-munittestdiscovertestsExampleLet's start with a toy model to learn the APIs. Firstly, you need to define apredict_batchmethod in your model class, and then add thebatchingdecorator to your model class.Thebatchingdecorator adds host() method to createModelHostobject. Thepredictmethod of ModelHost takes a single query as input, and it will merge multiple queries into a batch before callingpredict_batchmethod. The predict method also splits outputs from predict_batch method before it returns result.importnumpyasnpfrombatch_inferenceimportbatchingfrombatch_inference.batcher.concat_batcherimportConcatBatcher@batching(batcher=ConcatBatcher(),max_batch_size=32)classMyModel:def__init__(self,k,n):self.weights=np.random.randn((k,n)).astype("f")# shape of x: [batch_size, m, k]defpredict_batch(self,x):y=np.matmul(x,self.weights)returny# initialize MyModel with k=3 and n=3host=MyModel.host(3,3)host.start()# shape of x: [1, 3, 3]defprocess_request(x):y=host.predict(x)returnyBatcheris responsible to merge queries and split outputs. In this case ConcatBatcher will concat input tensors into a batched tensors at first dimension. We provide a set of built-in Batchers for common scenarios, and you can also implement your own Batcher. SeeWhat is Batcherfor more information.
Package: batching
BatchingBatching is a set of tools to format data for training sequence models.Installation$pipinstallbatchingExample usageExample script exists in sample.py# Metadata for batch info - including batch IDs and mappings to storage resouces like filenamesstorage_meta=StorageMeta(validation_split=0.2)# Storage for batch data - Memory, Files, S3storage=BatchStorageMemory(storage_meta)# Create batches - configuration contains feature names, windowing config, timeseries spacingbatch_generator=Builder(storage,feature_set,look_back,look_forward,batch_seconds,batch_size=128)batch_generator.generate_and_save_batches(list_of_dataframes)# Generator for feeding batches to training - tf.keras.model.fit_generatortrain_generator=BatchGenerator(storage)validation_generator=BatchGenerator(storage,is_validation=True)model=tf.keras.Sequential()model.add(tf.keras.layers.Dense(1,activation='sigmoid')model.compile(loss=tf.keras.losses.binary_crossentropy,optimizer=tf.keras.optimizers.Adam(),metrics=['accuracy'])model.fit_generator(train_generator,validation_data=validation_generator,epochs=epochs)LicenseMIT licenseCopyright 2015 ©FVCproductions.
Package: batch-ingestion-client-py
BatchIngestion Python ClientThis is a Python client library for theBatchIngestionMediaWiki extension, which provides an API to ingest many entities at once. This library allows you to easily ingest entities in bulk, either by parsing them from JSON or by creating them using Python objects.InstallationYou can install this library usingpip:pipinstallbatch-ingestion-client-py==1.0.4Usagefrombatch_ingestion_client_pyimport(BatchIngestor,Entity,ValueInLanguage,)ingestor=BatchIngestor(base_url="https://your-wiki.com",username="your-username",password="your-password",)example1=ingestor.ingest([Entity.parse({"type":"item","labels":{"en":{"language":"en","value":"Hello, world!",},},})])print(example1)example2=ingestor.ingest([Entity(type="item",labels={"en":ValueInLanguage(language="en",value="Hello, world!",),},),])print(example2)ContributingIf you'd like to contribute to this library, please feel free to submit a pull request.
Package: batch-mailchimp
A light wrapper aroundmailchimp-marketingthat makes it easier to use batch operations.Getting StartedInstallationpip install batch-mailchimpUsageThis can be used as a drop-in replacement for mailchimp-marketing –
just change the import at the top, and everything should work the same:importbatch_mailchimpasMailchimpMarketingclient=MailchimpMarketing.Client({"api_key":"YOUR_API_KEY",})The additional functionality comes when we initialise the client withbatch=True:importbatch_mailchimpasMailchimpMarketingclient=MailchimpMarketing.Client({"api_key":"YOUR_API_KEY","batch":True,})If we do this, operations are stored up in the client, to be run later. For example:# Fetch all MailChimp listsclient.lists.get_all_lists()All new operations will be added to the batch. When we’re ready, we can run all the operations in the batch:batch=batch_client.batches.run()We can check the batch’s status using:batch.status(refresh=True)Once it has finished, we can get the response with:response=batch.response()response.bodyAPI Structure and EndpointsThe API structure and endpoints match that ofmailchimp-marketing. You should refer to their documentation for usage.SupportIf you are having issues, pleasecreate an issue.LicenseThe project is licensed under the MIT License.
Package: batch-microsoft-translator
Batch Microsoft TranslatorBatch Microsoft Translator allows for large amounts of translations using the Microsoft 'Translator Text' API reliably.
Before starting a user must have a Microsoft Azure service account and must set up an account for the api to obtain a subscription key.
More information can be found here:https://azure.microsoft.com/en-gb/services/cognitive-services/translator-text-api/ReliabilityIf any fatal errors are found the translations that have been done are written to the output file.  When re run the process with check the output file and start from where it left off.
If any non fatal errors are found (e.g. internet connection issues) the process will wait and retry an number of times before raising a fatal error.Process TypesThe Microsoft 'Translator Text' API only allows for a mixumum of 25 strings of a with a maximum total length of 5000 characters.
The process will analyze each column in the data to find the best way to translate in order to reduce cost.  There are 2 processes that a column can be translated by:Batch Translations - Will go through each row and translate strings in batches, the process will work out the optimum batch size based on the average length of the strings found in each column.Unique Translations - If the columns contains a large number of repeated strings the process will not go through each row. This process will find all unique strings and only translate those then apply them back to each row.Parametersinput_df_or_path_or_buffer (DataFrame/str/StringIO) - the input can come from 1 of 3 sources:pandas DataFrame (DataFrame)input file path (str)a file buffer (StringIO)output_file_path (str) - the path where the output will be exported, or read from if needed.translate_cols_list (list) - the columns to translate in the data.subscriptionKey (str) - the subscription key from your Microsoft Azure service account.id_variable (str) - the process requires an id variable to exist in the data.tolang (str) - the language to translate to (default is "en"). More langauge codes can be found on the microsoft website.fromlang (str) - the language to translate to (default is None). If None the process will detect the language.delimiter (str) - the delimiter to use for teh input and output files (default is ",").translated_suffix (str) - the suffix to apply to the name of the translation columns (default is "_Translated").dtypes (dict) - any dtypes to apply when loading a input file (note all translation columns will be converted to str)capitalize (bool) - with clean text to 'proper' case if True (default is True)Examplefrombatch_microsoft_translatorimportrun_batch_translateinput_file="full_path"output_file="full_path"run_batch_translate(input_file,output_file,["Col1,Col2,Col3"],"XXXXXXXXXXXXXXXXXXXXXXXXXX","ID",dtypes={"ID":int})
Package: batchpath
AboutThis module is sort of like os.walk on steroids. It was created to allow the safe use of command-line path arguments in your Python scripts. For instance, say the user passes in several files and directories as arguments and you want to return a sorted list of files within said directories meeting specific access, extension, and/or size criteria. Or perhaps the user passes in a list of files and you want to verify that all of the files meet the necessary criteria before using them.Installationpip3 install batchpathThebatchpathmodule is known to be compatible with Python 3.NOTE: This module uses os.walk(), but will use scandir’s significantly faster implementation if it is available. Consider installing thescandirmodule.UsagefrombatchpathimportGeneratePaths()gp=GeneratePaths()paths=['/path/to/directory']files=gp.files(paths,access=os.R_OK,extensions=['conf','txt'],minsize=0,recursion=True)frombatchpathimportVerifyPaths()vp=VerifyPaths()paths=['/path/to/file','/path/to/dir','/path/to/other']verification_status=vp.all(paths,access=os.R_OK)invalid_paths=vp.failuresLicenseCopyright (c) 2015 Six (brbsix@gmail.com).Licensed under the GPLv3 license.
Package: batch-prediction-pipeline
Batch Prediction PipelineInstall for DevelopmentThe batch prediction pipeline uses the training pipeline module as a dependency. Thus, as a first step, we must ensure that the training pipeline module is published to our private PyPi server.NOTE:Make sure that your private PyPi server is running. Check theUsage sectionif it isn't.Build & publish thetraining-pipelineto your private PyPi server:cdtraining-pipeline
poetrybuild
poetrypublish-rmy-pypicd..Install the virtual environment forbatch-prediction-pipeline:cdbatch-prediction-pipeline
poetryshell
poetryinstallCheck theSet Up Additional ToolsandUsagesections to seehow to set uptheadditional toolsandcredentialsyou need to run this project.Usage for DevelopmentTo start batch prediction script, run:python-mbatch_prediction_pipeline.batchTo compute the monitoring metrics based, run the following:python-mbatch_prediction_pipeline.monitoringCheck out thisMedium articlefor more details about this module.NOTE:Be careful to set theML_PIPELINE_ROOT_DIRvariable as explained in thissection.