Saravanan666/pypi-org huggingface dataset:
Package: parallel-python-twitter
parallel-python-twitterA client that distributes Twitter API requests across multiple keys. Built for Python 3.6.Getting StartedFirst, get your API credentials together and get a list oftwitter.Apiobjects:import parallel_twitter
TWITTER_API_CONSUMER_KEY = ...
TWITTER_API_CONSUMER_SECRET = ...
OAUTHS = [
    {
        'oauth_token': ...,
        'oauth_token_secret': ...
    },
    {
        'oauth_token': ...,
        'oauth_token_secret': ...
    },
    ...
]
apis = parallel_twitter.oauth_dicts_to_apis(
    oauth_dicts=OAUTHS,
    api_consumer_key=TWITTER_API_CONSUMER_KEY,
    api_consumer_secret=TWITTER_API_CONSUMER_SECRET
)Next, try out some of the examples:user_ids = [
    561808704, # @neeljsomani
    813286, # @BarackObama
    17919972 # @taylorswift13
]
# Get a list of posts that these users liked
parallel_twitter.examples.pull_users_likes(
    users=user_ids,
    apis=apis
)
# Get a list of these users' posts + number of likes
parallel_twitter.examples.pull_users_posts(
    users=user_ids,
    apis=apis
)Comparison with TwintTwintis a Python library for scraping data from Twitter.From Twint'sdocumentation: "Twitter limits scrolls while browsing the user timeline. This means that with .Profile or with .Favorites you will be able to get ~3200 tweets." There are no such limits if you distribute your requests usingparallel-python-twitter.Twitter will block your requests if you scrape enough (ex:https://github.com/twintproject/twint/issues/682). I've testedparallel-python-twitterup to 100s of megabytes.Technically,twintviolates Twitter's Terms of Service, since scraping is not permitted in general.
Package: parallel-requests
parallel_requestsPerfomanceSeebenchmarksUsagePOST HTTP request usage:>>> import parallel_requests
>>> from typing import List
>>>
>>>
>>> responses: parallel_requests.ListResponse = parallel_requests.post(
>>>     method_args=[{"url": "https://www.python.org"}],
>>>     max_workers=10
>>> )
>>> responses
[<Response [200]>]GET HTTP request usage>>> import parallel_requests
>>> from typing import List
>>>
>>>
>>> responses: parallel_requests.ListResponse = parallel_requests.get(
>>>     method_args=[{"url": "https://www.python.org"}],
>>>     max_workers=10
>>> )
>>> responses
[<Response [200]>]request function usage:>>> import parallel_requests
>>> from typing import List
>>>
>>>
>>> responses: parallel_requests.ListResponse = parallel_requests.request(
>>>     method_args=[{"url": "https://www.python.org", "method": "post"}],
>>>     max_workers=10
>>> )
>>> responses
[<Response [200]>]
Package: parallel-rest-calls
Parallel REST CallsThe package provides a couple of methods that can be used to make parallel rest calls to retrieve multiple resources from a single REST collection.
This packages usesrequestspackage under the hood.UsageFor retrieving resources by pathparallelize = ParallelCalls(number_of_threads=20)
resource_ids = [2,3,5,8,13]

results = parallelize.path_calls('https://www.sample.com/items', resource_ids)For retrieving resources by query paramsparams = [1,2,3]
results = parallelize.path_calls('https://www.sample.com/items', "param_name" params)The return value is in the following format[
    {'status_code': 200, 'id': 'id_1', 'result': "result_object_1"},
    {'status_code': 200, 'id': 'id_2', 'result': "result_object_2"}
]Reporting BugsDevelopment and bug tracking is performed on GitHubContributingmake initinitializes the repo.make testsruns the tests.Feel free to raise a PR. Please ensure all tests pass before raising a PRLicenseMIT LicenseCopyright (c) 2022 Parallel REST CallsPermission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Package: parallels
parallelsParallels is aliterarypowered library.
It provides an async API that mimics concurrent.futures, with support for task-graph executors.These notebooks can be viewed usingnbvieweruntil the documentation generator is complete.What?In Python there are several standard APIs for interacting withexecutors. AsyncIO has therun_in_executorAPI,concurrent.futureshas theExecutorAPI, and other
libraries like Dask and Ray have equivalent approaches.concurrent.futuresis often available within other libraries, but its reduced features-set prevents the underlying
library from implementing useful optimisations like Dask's deferred computation or task graph building.Parallels implements a standardExecutorinterface which defines a synchronoussubmitmethod, and an asynchronousretrievemethod.
These methods operate upon value-lessasyncio.Futurehandles which yieldTrueupon task success, and raise anExceptionotherwise.
The Dask and Ray implementations accept these handles as arguments to futuresubmit()calls, which can be used to build task graphs
and avoid copying data to the local machine.
Package: parallel-sort
parallel-sortThis module provides a simple Cython interface to parallel sorting algorithms available in C++. It provides functions for in-place and out-of-place sorts, as well as indirect sorting (aka. "argsort").By default it requires a C++17-capable compiler, and will use the standardstd::sortroutine with the parallel execution policy introduced in that standard. For older compilers without C++17 support, theGNU Parallel Modesorting routines may be used instead.Usageimportparallel_sortimportnumpyasnpx=np.random.random(size=10000)x_sorted=parallel_sort.sort(x)assertnp.all(np.diff(x)>=0)# x is now sortedNote that these routines are "unstable" sorts, meaning that the ordering of equal elements in the original array is not guaranteed to be preserved.InstallingRequirements: numpy, C++17-capable g++, Cython (only for installation from source).On Linux, installing from wheel viapipshould "just work":pipinstallparallel_sortOn Mac, it is important to ensure that gcc is used to compile the module rather than the default Apple-provided clang, which can be done by prefixing the install command withCC=$(brew --prefix gcc)/bin/gcc-13(substituting your gcc version as appropriate).GNU Parallel Mode optionFor older compilers that do not support C++17, the module can use the GNU Parallel Mode library instead. To build without C++17, clone the repository and setuse_cxx17 = Falseinsetup.py. Then run:pipinstall-e.to compile and install the module.Note for Apple Silicon MacsIf the module builds OK, but importing it fails with an error "undefined reference toaarch64_ldadd4_acq_rel", try rebuilding with the following commandCFLAGS=-mno-outline-atomicsCC=$(brew--prefixgcc)/bin/gcc-13pipinstall--no-cache-dir-e.
Package: parallelsoup
ParallelSoupA library that utilizes BeautifulSoup to provide multi threaded scrapping, reducing the total time involved in the scrapping processAimThe library should implement the following affectively: (this list can be extended in future):Parallelization (I mean duh, you didn't see the repo name or what...)Should consider the fact that user already has serial BeautifulSoup codeAdding to above point: should be easier for user too just use our libraryAll parts of the library should be documentedAll parts of the library should have unit tests written for verification of their functionalityShowcase written examples for different sorts of scrappingExampleA small example of scraping imdb with ParallelSoup with goodness of 8 threadsfromparallelsoupimportParallelSoupurls=[]foriinrange(0,10):urls.append("https://www.imdb.com/search/title/?countries=in&adult=include&start="+str(50*i+1)+"&ref_=adv_nxt")defextractor(soup):data=[]foriteminsoup.findAll('div',attrs={'class':'lister-item mode-advanced'}):title=item.find('div',attrs={'class':'lister-item-content'}).h3.aiftitle!=None:data.append(title.text)returndataps=ParallelSoup(8,urls,extractor)ps.start()dataParallel=ps.get()print(dataParallel)IssuesThis library is still in development, for reporting any bugs or suggestions please open a new issuehere
Package: parallel-statistics
OverviewThis package collects tools which compute weighted statistics on parallel, incremental data, i.e. data being read by multiple processors, a chunk at a time.The tools available are:ParallelSumParallelMeanParallelMeanVarianceParallelHistogramSparseArrayAll assume that mpi4py is being used among the processes, and are passed a communicator object (oftenmpi4py.MPI.COMM_WORLD).InstallationFor now you can install this package using:pip install parallel_statisticsDocumentationDocumentation can be found athttps://parallel-statistics.readthedocs.io/ExampleThe three toolsParallelSum,ParallelMean, andParallelMeanVariancecompute statistics in bins, and you add data to them per bin.The usage pattern for them, andParallelHistogram, is:Create a parallel calculator object in each MPI processHave each process read in their own chunks of data and add it using theadd_datamethodsOnce complete, call thecollectmethod to get the combined results.Here's an example of splitting up data from an HDF5 file, using an example from the DESC tomographic challenge.  You can run it either on its own, or under MPI with different numbers of processors, and the results should be the same:importmpi4py.MPIimporth5pyimportparallel_statisticsimportnumpyasnp# This data file is available at# https://portal.nersc.gov/project/lsst/txpipe/tomo_challenge_data/ugrizy/mini_training.hdf5f=h5py.File("mini_training.hdf5","r")comm=mpi4py.MPI.COMM_WORLD# We must divide up the data between the processes# Choose the chunk sizes to use herechunk_size=1000total_size=f['redshift_true'].sizenchunk=total_size//chunk_sizeifnchunk*chunk_size<total_size:nchunk+=1# Choose the binning in which to put valuesnbin=20dz=0.2# Make our calculatorcalc=parallel_statistics.ParallelMeanVariance(size=nbin)# Loop through the dataforiinrange(nchunk):# Each process only reads its assigned chunks,# otherwise, skip this chunkifi%comm.size!=comm.rank:continue# work out the data range to readstart=i*chunk_sizeend=start+chunk_size# read in the input dataz=f['redshift_true'][start:end]r=f['r_mag'][start:end]# Work out which bins to use for itb=(z/dz).astype(int)# add add each oneforjinrange(z.size):# skip inf, nan, and sentinel valuesifnotr[j]<30:continue# add each data pointcalc.add_datum(b[j],r[j])# Finally, collect the results togetherweight,mean,variance=calc.collect(comm)# Print out results - only the root process gets the data, unless you pass# mode=allreduce to collect.  Will print out NaNs for bins with no objects in.ifcomm.rank==0:foriinrange(nbin):print(f"z = [{dz*i:.1f}..{dz*(i+1):.1f}]    r ={mean[i]:.2f}±{variance[i]:.2f}")
Package: paralleltools
Parallel tools (named in manner to itertools & functools) is a set of commonly used list traversal functions, which is working in parallel (fault-tolerant) in synchronous or asynchronous manner.Implementation is based on pythonthreadingmodule, so be aware of GIL.Currently implemented functions are (both sync & async):filter- filters the list by predicate you provide;map- applies a function to each element of the list.Important: Due to nature of parallel processing the order of results isn’t guranteed. Although, function is returns alistbecause the objects you want to process might not be hashable, hence you can’t use aset.
Package: parallel-transform
parallel-transformA implementation of asynchronous multiprocessing with progress logging.Installationpipinstallparallel-transformFunctions:transform_list_parallel(
list_to_transform,
transform_func,
*transform_args,
verbose=True,
processes=-1,
min_log_frequency=1000,
logger=None,
log_func='info')The elements inlist_to_transformwill be transformed based on thetransform_funcfunction, with the list element as the first arg, and*transform_argsas the remaining args. If verbose is set to true, the progress every 5% ormin_log_frequencyprocesses will be logged, whichever is less. An optional logger can be passed; if no logger is passed, the logs will go to std out.The order of the list_to_transform is maintained.
Package: parallel_websocket_server
parallel_websocket_serverrun websocket server in parallel (using multiprocessing)You can run the websocket server as a subprocess while keeping the main process distinct.You can implement the main process without having to pay attention to the server.Installationpipinstallparallel_websocket_serverHow to useimportjsonimportrandomimportmultiprocessingasmpfromparallel_websocket_serverimportParallelWebSocketServerif__name__=="__main__":receive_queue=mp.Queue()send_queue=mp.Queue()server=ParallelWebSocketServer(receive_queue,send_queue)whileTrue:ifnotreceive_queue.empty():received_message=receive_queue.get_nowait()print(received_message)ran=random.random()ifran<0.00001:send_queue.put(json.dumps({"hoge":1}))
Package: parallelworkloads
A Python port of the Workload Model proposed by Lublin & Feitelson.The following code shows how to use it:fromparallelworkloadsimportlublin99w=lublin99.Lublin99(1,2)# Will use both batch and interactive jobsw.numJobs=4# will generate four jobsw.generate()# The four generated jobs are shown below[SwfJob(jobId=1,submissionTime=103,waitTime=-1,runTime=12379,allocProcs=16,avgCpuUsage=-1,usedMem=-1,reqProcs=-1,reqTime=-1,reqMem=-1,status=1,userId=-1,groupId=-1,executable=-1,queueNum=1,partNum=-1,precedingJob=-1,thinkTime=-1),SwfJob(jobId=2,submissionTime=3089,waitTime=-1,runTime=177,allocProcs=16,avgCpuUsage=-1,usedMem=-1,reqProcs=-1,reqTime=-1,reqMem=-1,status=1,userId=-1,groupId=-1,executable=-1,queueNum=1,partNum=-1,precedingJob=-1,thinkTime=-1),SwfJob(jobId=3,submissionTime=3150,waitTime=-1,runTime=10,allocProcs=2,avgCpuUsage=-1,usedMem=-1,reqProcs=-1,reqTime=-1,reqMem=-1,status=1,userId=-1,groupId=-1,executable=-1,queueNum=0,partNum=-1,precedingJob=-1,thinkTime=-1),SwfJob(jobId=4,submissionTime=3172,waitTime=-1,runTime=7,allocProcs=32,avgCpuUsage=-1,usedMem=-1,reqProcs=-1,reqTime=-1,reqMem=-1,status=1,userId=-1,groupId=-1,executable=-1,queueNum=0,partNum=-1,precedingJob=-1,thinkTime=-1)]User runtime estimatesparallelworkloadsalso supports generating runtime estimates based on the
model proposed by Dan Tsafrir in 2005. For the model to work, it needs at least
200 jobs. Here’s an example continuing the previous one:fromparallelworkloadsimporttsafrir05w.numJobs=200jobs=w.generate()print('Original requested time of first job:',jobs[0].reqTime)t=tsafrir05.Tsafrir05(jobs)jobs=t.generate(jobs)print('Generated requested time of first job:',jobs[0].reqTime)Which gives as output:Original requested time: -1
Generated requested time: 22962.0
Package: parallel-write
SummaryParallel write is a Python module for distributing writes between an arbitrary number of open
file(like) objects.Features:Distributes each calls to the proxy object to each passed file objects, so all of them
should be in the same stateWrites are done in a configurable length thread pool, so you can have slower underlying
objects, their slowness won’t add upCompares results from the methods, so despite its name, you can actually read from many objects
at once and fail if any of them return different dataMotivationWe often write the same data to local disk (for later caching) and remote (S3 for persistence).
The files must be the same, but the tool we’re using may produce binary-different outputs for
two subsequent writes (either becausePYTHONHASHshuffles things or it includes time-stamps
into the compressed output’s metadata, doesn’t matter).We could write the file locally first, then copy it to S3, but that would take more time and
complexity in code. It’s easier to write them at the same time.How to use this moduleSee thedocumentation.Issues and DiscussionsAs usual in any GitHub based project, raise anissueif you find any bug or room for
improvements.Versionv0.0.10Changelogv0.0.10 (2022-01-14)v0.0.9 (2022-01-03)v0.0.8 (2022-01-03)v0.0.7 (2021-12-16)v0.0.6 (2021-12-13)v0.0.5 (2021-12-13)initial version
Package: parally
parallyA python package to distribute CPU-intensive tasks by sending workload to other connected computersStatusDocumentationThe documentation for this package can be foundhere.UsageTo use the package, run the following command in the terminal:pipinstallparallyDevelopmentTo install the package, make sure conda is installed and then run the following commands in the terminal:# Clone the repositorygitclonehttps://github.com/acse-ci223/parally.git# Change directorycdparally# Create the 'parally' environmentcondaenvcreate-fenvironment.yml# Activate the environmentcondaactivateparally# Install the packagepipinstall-e.ExamplesExample 1: Distributing a simple taskserver.pyfromparallyimportServerHOST="localhost"PORT=5000parameters=[{"a":1,"b":2},{"a":3,"b":4},{"a":5,"b":6}]defcompleted_task(result):print(result)server=Server(HOST,PORT)server.bind_parameters(parameters)server.on_completed(completed_task)server.on_error(lambdaerror:print(error))server.start()client.pyfromparallyimportClientHOST="localhost"PORT=5000defmy_function(params):a=params['a']b=params['b']returna+bclient=Client(HOST,PORT)client.run_function(my_function)client.start()
Package: paraloop
Automatically parallelizing simple python for-loopsThis is currently still in a pretty experimental stage, and I can guarantee that it is full of horrendous bugs. There are only twoAggregationStrategytypes with support for a limited set of objects for now, but the library can definitely be used (especially for loops that don't "return" any values at all). Have a look at theaggregation strategiesto see what is currently supported!How it worksGiven a for-loop, e.g.fromparaloopimportParaLoop,Variablefromparaloopimportaggregation_strategiescounter=Variable(3,aggregation_strategy=aggregation_strategies.Sum)dictionary=Variable({},aggregation_strategy=aggregation_strategies.Concatenate)foriinParaLoop(range(0,100),num_processes=8):counter+=idictionary[f"key_{i}"]="Hi!"print(counter)print(dictionary)paraloopwill turn it into a function, e.g.defloop_iterator(i):counter.assign(counter+1)dictionary[f"key_{i}"]="Hi"!And will call the function once for every iteration of the loop across multiple processes, instead of the original loop body.
Once the processes have finished,paraloopwill handle the aggregation based on the chosenAggregationStrategy, so that you can access your variable as if no multiprocessing ever happened.When would I use this?paraloopis intended to be used for parallelizing for-loops that take an annoying amount of time, but are not worth spending the time and effort of proper multiprocessing on. These are usually fairly simple loops in research-style code that involve many web or file operations, but the goal ofparaloopis to support parallelizinganyPython for-loop by simply wrapping the variables and callingParaLoop, without other modifications to the source code.paraloopisnotintended to be optimally efficient or provide a robust multiprocessing framework, and you probably shouldn't want to use this in a production environment. If you're looking for a robust multiprocessing framework that does require a bit of setup (i.e. rewriting your loop to a function with some specific return value and then aggregating those values yourself), have a look atjoblib.Practical exampleHave a look atexample.py.
It queries some WikiPedia pages and counts the frequency of each word.
The output of the script is as follows:13882 57495
The original loop took 18.553406238555908 seconds.
13882 paraloop.Variable(57495)
The ParaLoop took 1.689873456954956 seconds.Which is of course because most of the time is spent waiting for the WikiPedia server to respond.RoadmapWrite unit tests for theParaLoopclass and the loop transformerAutomatically determine the optimal number of processes if none was specifiedAdd an optional progress barAdd a timeout in case a worker silently failsAddSharedVariables that are stored in shared memory and hence don't need to be aggregated at all
Package: paralytics
Documentation:https://mrtovsky.github.io/Paralytics/.Repository:https://github.com/mrtovsky/Paralytics/.What is it?Paralyticspackage was created in order to simplify and accelerate repetitive
tasks during modeling and predictive analysis. It especially puts stronger emphasis
on data preprocessing, which is often the most arduous stage of modeling.The purpose of this package is to reduce to a minimum time allocated on repetitive
activities preceding the problem-specific approach to a given problem, containing
among others optimization of the applied machine learning techniques, which is the
part that most of Data Scientists would like to devote the most energy to, however,
by poorly prepared data, it is often only a fraction of the total work time devoted
to the project.Main FeaturesHighlighting the main functionalities of theParalytics:Expandedtarget encodingof categorical variables using double cross-validation
technique with additional regularisation preventing favoritism of sparse categories
with reduction of excessive adjustment to the training set, effectively reducing
overfitting.Discretizationof continuous variables to ordinal using shallow decision tree or
method based on Spearman’s rank-order correlation.Processing data read into theDataFrames,
including:automatic unification of variable types,grouping of sparse categories,projecting of text variables whose unique elements symbolize a binary response onto
binary variables,imputation of missing data.Collinearity reduction using such factors as:variance inflation factor(VIF) or correlation.InstallationDependenciesParalyticspackage requirements are checked and, if needed, installed during the installation
process automatically. Mainly used packages across theParalyticsare:NumPy(>= 1.15.4)Pandas(>= 0.23.4)scikit-learn(>= 0.20.1)SciPy(>= 1.1.0)For visualizations:matplotlib(>= 3.0.2)The easiest way to install the package is usingpip:pip install paralyticsIf you want to omit the packages upgrade (when your version does not meet the requirements) run:pip install paralytics --no-depsWhen functionalities requiring optional dependencies are needed you can install those extra requirements by running:pip install paralytics[<extra-name>]For example, to useBaseSeleniumBrowser(requires selenium) andVIFSelector(requires statsmodels) run:pip install paralytics[browser,vif]Installation of all extras is possible via:pip install paralytics[all]You can always install directly from the githubrepository:pip install git+https://github.com/mrtovsky/Paralytics.git
Package: param
Build StatusCoverageLatest dev releaseLatest releasePythonDocsBinderSupportParam is a library providing Parameters: Python attributes extended to have features such as type and range checking, dynamically generated values, documentation strings, default values, etc., each of which is inherited from parent classes if not specified in a subclass.Param contains only two required Python files, with no external dependencies, and is provided freely for both non-commercial and commercial use under a BSD license, so that it can easily be included as part of other projects.Please seeparam's websitefor official releases, installation instructions, documentation, and examples.