Saravanan666/pypi-org huggingface dataset:
Package: redpajama
RedPajama-Data: An Open Source Recipe to Reproduce LLaMA training datasetThis repo contains a reproducible data receipe for the RedPajama data, with the following token counts:DatasetToken CountCommoncrawl878 BillionC4175 BillionGitHub59 BillionBooks26 BillionArXiv28 BillionWikipedia24 BillionStackExchange20 BillionTotal1.2 TrillionData PreparationIndata_prep, we provide all pre-processing scripts and guidelines.TokenizationIntokenization, we provide an example of how to tokenize the dataset using the GPT-NeoX tokenizer.VisualizationInviz, we provide a dashboard for exploring a subset of the data usingMeerkat.LicenseThe code in this repo is licensed under the Apache 2.0 license. Unless otherwise noted,Copyright 2023 Together Computer, ETH Zürich, Stanford University

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.The filedata_prep/book/dedup.pywas co-developed withOntocord.ai.Copyright 2023 Ontocord.ai, Together Computer, ETH Zürich, Stanford University

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.The dataset itself, please refer to the licenses of the data subsets you use.Common Crawl Foundation Terms of UseC4 licenseGitHub was limited to MIT, BSD, or Apache licenses onlyBooks:the_pile_books3 licenseandpg19 licenseArXiv Terms of UseWikipedia LicenseStackExchange license on the Internet ArchiveFor full terms, see the LICENSE file. If you have any questions, comments, or concerns about licensing pleasecontact us.AcknowledgementWe are appreciative to the work done by the growing open-source AI community that made this project possible. That includes:Participants in building the RedPajama dataset includingOntocord.ai,MILA Québec AI Institute,ETH DS3Lab,Université de Montréal,Stanford Center for Research on Foundation Models (CRFM),Stanford Hazy Research research groupandLAION.EleutherAI— This project is built on the backs of the great team at EleutherAI — including the source code they provided for training GPT-NeoX.An award of computer time was provided by theINCITE program. This research also used resources of theOak Ridge Leadership Computing Facility (OLCF), which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725.
Package: redpajama-data
RedPajama-Data: An Open Source Recipe to Reproduce LLaMA training datasetThis repo contains a reproducible data receipe for the RedPajama data, with the following token counts:DatasetToken CountCommoncrawl878 BillionC4175 BillionGitHub59 BillionBooks26 BillionArXiv28 BillionWikipedia24 BillionStackExchange20 BillionTotal1.2 TrillionData PreparationIndata_prep, we provide all pre-processing scripts and guidelines.TokenizationIntokenization, we provide an example of how to tokenize the dataset using the GPT-NeoX tokenizer.VisualizationInviz, we provide a dashboard for exploring a subset of the data usingMeerkat.LicenseThe code in this repo is licensed under the Apache 2.0 license. Unless otherwise noted,Copyright 2023 Together Computer, ETH Zürich, Stanford University

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.The filedata_prep/book/dedup.pywas co-developed withOntocord.ai.Copyright 2023 Ontocord.ai, Together Computer, ETH Zürich, Stanford University

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.The dataset itself, please refer to the licenses of the data subsets you use.Common Crawl Foundation Terms of UseC4 licenseGitHub was limited to MIT, BSD, or Apache licenses onlyBooks:the_pile_books3 licenseandpg19 licenseArXiv Terms of UseWikipedia LicenseStackExchange license on the Internet ArchiveFor full terms, see the LICENSE file. If you have any questions, comments, or concerns about licensing pleasecontact us.AcknowledgementWe are appreciative to the work done by the growing open-source AI community that made this project possible. That includes:Participants in building the RedPajama dataset includingOntocord.ai,MILA Québec AI Institute,ETH DS3Lab,Université de Montréal,Stanford Center for Research on Foundation Models (CRFM),Stanford Hazy Research research groupandLAION.EleutherAI— This project is built on the backs of the great team at EleutherAI — including the source code they provided for training GPT-NeoX.An award of computer time was provided by theINCITE program. This research also used resources of theOak Ridge Leadership Computing Facility (OLCF), which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725.
Package: redpanda
RedPanda: Pandas & SQLAlchemyTwo great tastes that taste great together.Use RedPanda to add simple pandas integration into your declarative models.InstallationpipinstallredpandaBasic UseCreate a session from a SQLAlchemy engine:importredpandaengine=redpanda.create_engine("sqlite://")# => Engine(sqlite://)Session=redpanda.orm.sessionmaker(bind=engine)session=Session()# => <sqlalchemy.orm.session.Session>QueryingUse theframe()method of RedPanda queries to return a DataFrame representation of the results instead of a collection of models.query=session.query(MyModel)# => <redpanda.orm.Query>query.frame()# => <pandas.DataFrame>Querying with FiltersTheframe()method that wraps thepandas.read_sql()function into a dialect-agnostic class-method for declarative SQLAlchemy models and can accept the same keyword arguments aspandas.read_sql():query=session.query(MyModel).filter_by(my_attr="my_val")query.frame(index_col="time")Additionally, awithin()method is added to SQLAlchemy's InstrumentedAttribute class that accepts a pandas Index object:index=pandas.period_range("2016-11-01","2016-11-30",freq="W")query=session.query(MyModel).filter(MyModel.timestamp.within(index))
Package: redparser
RedparserRedparser is a simple Python program that can parse a log based on command line arguments.UsageRunutil.pyas a Python script, passing the log file as the last argument to utilize
the mmap library for fast searches or push the log file to memory for the util to read log file from stdin.Example:./util.py--last10--timestamps--ipv4--ipv6example.logOutputs:The complete usage can be seen by using the -h flag.usage: ./util.py [OPTION]... [FILE]
usage: cat [FILE] | ./util.py [OPTION]
usage: redparser [OPTION]... [FILE]
usage: cat [FILE] | redparser[OPTION]

Parse logs of various kinds

optional arguments:
  -h, --help            show this help message and exit
  -f, --first NUM       Print first NUM lines
  -l, --last NUM        Print last NUM lines
  -t, --timestamps      Print lines that contain a timestamp in HH:MM:SS format
  -i, --ipv4 IP         Print lines that contain an IPv4 address, matching IPs (Optional) highlighted
  -I, --ipv6 IP         Print lines that contain an IPv6 address, matching IPs (Optional) highlightedInstall with pippip install redparserTODOSearch by timestampShould be able to check ip address from cmdline matches a line (easy fix)
Package: redpic
Relativistic Difference Scheme Particles-In-Cell (REDPIC)This PIC code has been developed since 2022 as an alternative toASTRA,WARP,XTRACKand other codes.For particle dynamics simulation using finite difference scheme relativistic.Table of contentGetting StartedLocal build and launchDocumentationAdding a new sectionAdditional resourcesCUDA installationPrerequisitesLatexUseful ResourcesIDEAuthorsLicenceContributingDependenciesFormattingTestsMaintainingGetting StartedLocal build and launchTo build our application and create a Docker image, it will be enough to run the following command:docker build -t redpic .To launch the application, use the command:docker run -it redpicDocumentationThe documentation contains all the Latex files needed to generate documentation.
The main source files are located in themanual.main.texis documentation source file.
The final PDF can be foundhere.main.texis the source file that Latex compiler will use to generate the paper.
However, in order to keep the code cleaner, the main sections of the paper are all located in thesections.
In this way you will experience less merging issues when two or more people are working on the same doc.Just edit the text in the relative Latex file (e.g., introduction, methodology, etc.) and you should be ready to go.
No need to change any other file.Adding a new sectionJust copy a section file (e.g.,introduction.tex) paste it in the same directory.
Rename the pasted file (e.g. first_chapter.tex) and add this file tomain.tex.Additional ResourcesAlternatively you can find great resources on theOverleaf Tutorial websiteor onLatex wikibooks.CUDA installationFor CUDA implementation we useNumbalibrary.
For more information about CUDA installation seeNumba CUDA installation guide.PrerequisitesLatexLatex IDE and compiler installed locally on your machine.
We recommend using a PyCharm plugin calledTeXiFy IDEAas IDE andmiktexas Latex compilerAlternatively you can push your code to Overleaf using git and only use Overleaf.
We would discourage you from doing this! Overleaf should only be used for the review.Useful ResourcesIDEYou may want to take advantage of the power of IDEs.
For Python We would recommend usingPyCharm.Alternatives are:Visual StudioAtomGitYou should installgiton your computer. And haveGitHubaccount.AuthorsVyacheslav Fedorov-Initial workDanila Nikiforov-Initial workLicenseThis project is licensed under the MIT License - see theLICENSEfile for detailsContributingDependenciesUsemake depscommand to install library, its production and development dependencies.FormattingUsemake formatto autoformat code with black tool.LinterUsemake lintto run only linters for current python versionTestUsemake testto run test for current python versionMaintainingIf pull request consists of several meaningful commits, that should be preserved,
then use "Rebase and merge" option. Otherwise use "Squash and merge".New release (changelog, tag and pypi upload) will be automatically created
on each push to master via Github Actions workflow.
Package: redpie
RedpieUse redis just like a Python dictionaryHow to useInstallation$ pip install redpieUsagefrom redpie import Redpie

r = Redpie(db=0, host='localhost', port=6379)  # Or just r = Redpie(); 0, localhost and 6379 are defaults


from requests import Session

r.update({
    'you can': 'assign basic types',
    'and even more complex things': Session
})

>>> print(r['and even more complex things']())
<requests.sessions.Session object at 0x10c68d5c0>
Package: redpill
Redpill is a cross-platform package management framework that lets you easily
manage and build dependencies for large projects.FeaturesUsageUsage: redpill <command> [options]

Commands:

    build      download and build the dependencies
    info       show metadata relating to the installs
    install    install specific build packages
    nuke       nuke the local install
    uninstall  uninstall specific build packages

    check      check if a repo checkout is up-to-date
    version    show the version number and exit

See `redpill help <command>` for more info on a specific command.ContributeTo contribute any patches simply fork the repository using GitHub and send a
pull request tohttps://github.com/tav, thanks!LicenseAll of the code has been released into thePublic Domain. Do with it as you
please.–
Enjoy, tav <tav@espians.com>
Package: redplanet
RedPlanetRedPlanet is a Python package that gives an easy way to work with various Mars datasets. With straightforward methods and high customizability, you can either create publication-ready plots on the fly, or access the underlying data for more involved calculations.Usage:Install withpip install redplanet.GRSAccess/visualize chemical abundance maps derived from the 2001 Mars Odyssey Gamma Ray Spectrometer. The original data is defined in 5 degree bins, but this module allows you to programmatically estimate values at exact coordinates by interpolating between points. Concentrations can be extracted for both the shallow subsurface (raw data) and bulk crustal composition (normalized to volatile-free basis, i.e. zero H2O/Cl/Si).Example 1: Iron concentrations in Arabia Terra. Data is normalized to a volatile-free basis (H2O/Cl/Si free), so it is representative of the bulk crustal composition rather than shallow subsurface.Recreate with:fromredplanetimportGRSGRS.visualize(element_name='fe',normalize=True,lon_bounds=(5,40),lat_bounds=(-10,30),grid_spacing=0.1,overlay=True)CrustAccess/visualize maps of topography, Moho, crustal thickness, and crustal density from spherical harmonics. We offer ~22,000 models of the crust-mantle interface parameterized by reference interior models, crustal thickness at the InSight landing, and homogeneous/inhomogeneous crustal densities across the dichotomy.Example 1: Topography, Moho, and crustal thickness of the Valles Marineris region. Model parameters are 41 km crustal thickness at the InSight landing, 2,900 kg/m^3 crustal density in the North, and 2,700 kg/m^3 crustal density in the South.Recreate with:fromredplanetimportCrustlons=(-100,-20)lats=(-60,40)Crust.load_model(RIM='Khan2022',insight_thickness=41,rho_north=2900,rho_south=2700)Crust.visualize(quantity='topo',lon_bounds=lons,lat_bounds=lats,grid_spacing=0.1)Crust.visualize(quantity='moho',lon_bounds=lons,lat_bounds=lats,overlay=True,grid_spacing=0.3)Crust.visualize(quantity='thick',lon_bounds=lons,lat_bounds=lats,overlay=True,grid_spacing=0.3)Example 2: Crustal thickness profile of Henry Crater with various crust-mantle interface models.Recreate with: See section 2.2 indemo.ipynb.DocumentationFor a more in-depth tutorial in interactive notebook format, seedemo.ipynb.LinksDon't hesitate to reach out:zain.eris.kamal@rutgers.eduAcknowledgements:docs/thanks.txtReferences:docs/references.txtOther work:github.com/Humboldt-Penguin
Package: red-py-sdk
red-py-sdkinstallpip3 install red-py-sdkUsageImportfrom redpysdk import ReddioInit objectInit the object, you can use 'testnet' or 'mainnet' to init the sdkreddio = Reddio("testnet")Get StarkKey PairUsageget_stark_key_pair()It will generate an random starkkey pairExample>>> reddio.get_stark_key_pair()
('0x395d1708ab0ee91efcb7f26a2f4fcbe20faf3c7390517667fed37b0e481882a', '0x5aa1b67a486b6564a2b6ae7426950c03dfe6991f9d34ea45b6b0be0672a1818')Mint NFTUsagemintNFT(api_key, contract_address, stark_key, amount)It will mint $amount tokens of $contract_address to $stark_key. first you need to register at dashboard(dashboard.reddio.com) to get your $api_key, and make sure you register the contract_address as ERC721MExample>>> reddio.mintNFT(your_api_key,"0xd60523fd920eb9b7eff3e115203e32d91de5cf59","0x1ccc27877014bc1a81919fc855ebbd1b874603283c9ea93397d970b0704e581","10")
[302808, 302809, 302810, 302811, 302812, 302813, 302814, 302815, 302816, 302817]Get BalanceUsageget_balances(stark_key, page=1, limit=10)It will return the starkkey's balance. including ERC20/ETH/ERC721Examplereddio.get_balances("0x6ecaebbe5b9486472d964217e5470380782823bb0d865240ba916d01636310a")TransferUsagetransferNFT(stark_private_key, starkkey, receiver, token_type, contract, tokenID, expiration_timestamp=4194303)parametersstark_private_key: The private key of layer2starkkey: The stark key of layer2receiver: The receiver, should be starkkey of other accounttoken_type: ERC721 or ERC721M. if mint on layer2 then it should be ERC721M. else it should be ERC721tokenID: The token idexpiration_timestamp: When will it expiration, it is unix timestamp/3600Examplereddio.transferNFT('private_key', '0x6ecaebbe5b9486472d964217e5470380782823bb0d865240ba916d01636310a', '0x1ada455b26b246260b7fd876429289639d7a0ce5fe295ff2355bd4f4da55e2', 'ERC721', '0x941661Bd1134DC7cc3D107BF006B8631F6E65Ad5', '618'))In the example, you should replace the 'private_key' to the private key of the starkkeyWithdrawalUsagewithdrawNFT(stark_private_key, starkkey, receiver, token_type, contract, tokenID, expiration_timestamp=4194303)parametersstark_private_key: the private key of layer2starkkey: the stark key of layer2receiver: the receiver, should be starkkey of other accounttoken_type: ERC721 or ERC721M. if mint on layer2 then it should be ERC721M. else it should be ERC721tokenID: the token idexpiration_timestamp: when will it expiration, it is unix timestamp/3600Examplereddio.withdrawNFT('private_key', '0x6ecaebbe5b9486472d964217e5470380782823bb0d865240ba916d01636310a', '0xffc882996cFAB2C8B9983394E09bb025a98e52bc', 'ERC721', '0x941661Bd1134DC7cc3D107BF006B8631F6E65Ad5', '663')In the example, you should replace the 'private_key' to the private key of the starkkeyOthersbuy_nftsell_nft
Package: redque
A distributed queue based on RedisInstallationpipinstallredqueExampleimporttimeimportrandomimportthreadingimportredisimportredquedefproductor(redis_conn):other_queue=redque.Queue(redis_conn,"__test","bp__test")for_inrange(100):other_queue.push("e_{0}".format(random.randint(1,100)))time.sleep(0.1)if__name__=="__main__":pool=redis.ConnectionPool(host="127.0.0.1",port=6379,password="123456")queue=redque.Queue(redis.Redis(connection_pool=pool),"__test","bp__test")queue.clear(True)# non-blockingfor_inrange(100):queue.push(random.randint(1,100))queue.process(lambdax:(print(x),True),False)# blockingt=threading.Thread(target=productor,args=(redis.Redis(connection_pool=pool),))t.start()queue.process(lambdax:(print(x),True),timeout=3)t.join()
Package: redquill
redquillLogging for pandas and more.DescriptionThere are functions specifically to log pandas dataframes as well
as functions to count up the total number of errors and warnings.Dependenciespandas: Tested on 0.20.3
and higher.  May work for earlier versions.ExampleFor this example, import both the customlogger and pandas.importredquillasrqimportpandasaspdInitialize the logger.  Note, the logger always logs to the console
by default. You may inrqude a log directory and the log file name,
which will allow the logs to flow into a file as well.log=rq.redquill()The following will log any null values found indf.df=pd.DataFrame({"A":[1,2,None,3],"B":[4,2,2,5]})log.warn_null_values(df=df)Console output.2018-03-04 01:34:33,162 - 23512 - WARNING - test_redquill.test_warn_null_values - 
    A  B
2 NaN  2The following will log any duplicates found in columnBofdf.df=pd.DataFrame({"A":[1,2,None,3],"B":[4,2,2,5]})log.warn_duplicate_values(df=df,subset="B",msg="Duplicates on B.")Console output.2018-03-04 02:29:41,419 - 31825 - WARNING - test_redquill.test_warn_duplicate_values - Duplicates on B.
     A  B
1  2.0  2
2  NaN  2
Package: redrawing
DocumentationredrawingPython package provinding tools for artistic interactive applications using AICreated by ReDrawing Campinas team for the OpenCV AI 2021 Competition.Installationpython -m pip install --upgrade pip
python -m pip install redrawingStagesStages are the basic processing units of the redrawing package. They receive data objects from input channels, processes them and output data in the output channels.Examples of stages:Stage nameUseOAK_StageHandles a OAK camera. Can be changed using OAK_Substages, like a Body detector or a Hand gesture detectorPCR_ViewerVisualizer for image, depth and body pose dataCameraReceiver/IMUReceiverReceives data from a smarthphoneUKF_IMUUKF filter for orientation estimation using IMU dataData and communicationAll inter stage data communication occours using Data classes.Examples of data classes:Data classUseBodyPoseStores body pose data (keypoints)ImageStores image dataDepth_MapStores depth dataIMUStores IMU dataFor exchange data with client applications, the UDP_Stage can be used, using UDP protocol with JSON converted messages. We also have client libraries for receiving and decoding data:redrawing_java: client library for JavaReDrawing for Processing: client library for Processing, encapsulates the redrawing_java libraryAuthorsArtemis Sanchez MoroniCássio Gião DezzottiDaniele Souza GonçalvesElton Cardoso do NascimentoGabriel Tadao KuaeJônatas ManzolliMarcela MedicinaPedro Victor Vieira de PaivaTiago Fernandes TavaresThiago Danilo Silva de LacerdaLegalCopyright (c) 2021 ReDrawing Campinas AUTHORSSubmodules in ai_models/third_models submodules may be licensed under other licenses. A copy of the respective license can be found in the submodule folder.Made with the support of "Conselho Nacional de Desenvolvimento Científico e Tecnológico" (CNPq) of Brazil