Saravanan666/pypi-org huggingface dataset:
Package: arxiv2bib
Provides a command line tool to get metadata for an academic paper
posted at arXiv.org in BibTeX format.Transform this:$ arxiv2bib 1001.1001Into this:@article{1001.1001v1,
Author        = {Philip G. Judge},
Title         = {The chromosphere: gateway to the corona,
  or the purgatory of solar physics?},
Eprint        = {1001.1001v1},
ArchivePrefix = {arXiv},
PrimaryClass  = {astro-ph.SR},
Abstract      = {I outline curious observations which I personally
find puzzling and deserving of attention.},
Year          = {2010},
Month         = {Jan}
}
Package: arxiv2kindle
arxiv2kindleA simple script to recompile arxiv papers to kindle-like format.How does it work?This script downloads the LaTeX source from arxiv
and re-compiles it trying to fit a smaller size.
We also apply some simple transforms such as:downsize images;add automatic line breaks in formulas;allow formulas be placed on the next line;try to convert two-sided documents to one-sided format.All these transformations are automatic, so the success is not guaranteed.
This approach will also not work for papers without the source.
Nevertheless, in most cases the result is readable
(tested on an old 6.5in x 4.5in Kindle).UsageWith your paper of choice run:arxiv2kindle --width 4 --height 6 --margin 0.2 1802.08395 - > out.pdforarxiv2kindle --width 6 --height 4 --margin 0.2 --landscape "Towards end-to-end spoken language understanding" ./Installationarxiv2kindlerequirespipversion 10.0 or greater.To install the package, runpip install arxiv2kindleAcknowledgementsThis script is based on this amazingnotebook.Related projectshttps://github.com/cerisara/arxiv2kindlehttps://knanagnostopoulos.blogspot.com/2013/03/from-arxiv-directly-to-my-kindle_15.htmlhttps://dlmf.nist.gov/LaTeXML/
Package: arxiv2speech
Note: this only works on Mac OS X.Inspired by a hack suggested at#hackAASbyKathy Cooksey.Built byDan Foreman-Mackeyand distributed under
the BSD 2-clause license (seeLICENSE).This web scraping project makes use of thehtml2textmodule written by one of the
greatest scrapers that that community has known:Aaron Swartz.InstallationYou can install usingpipwith:pip install arxiv2speechUsageUsage: arxiv2speech [-h | --help] [-u URL] [-o OUTPUT] [--clobber]

-h --help  show this
-u URL     custom arXiv RSS [default: http://export.arxiv.org/rss/astro-ph]
-o OUTPUT  base directory for output [default: ./build]
--clobber  overwrite an existing archive
--quiet    don't print any status messages
Package: arxiv2text
Development Status :: 3 - Alphaarxiv2textA Python package that converts arXiv documents into structured text using arXiv PDF URLs. If it's a PDF link other than arXiv, it's mostly feasible. If extraction to markdown fails, try extracting to text or HTML.Install$ pip install arxiv2text$ pip install git+https://github.com/dsdanielpark/arxiv2text.gitUsagearxiv_to_textfromarxiv2textimportarxiv_to_textpdf_url="https://arxiv.org/pdf/1706.03762"extracted_text=arxiv_to_text(pdf_url,output_dir)# output_dir is optional argumentarxiv_to_htmlfromarxiv2textimportarxiv_to_htmlpdf_url="https://arxiv.org/pdf/1706.03762"output_dir="output_folder"arxiv_to_html(pdf_url,output_dir)arxiv_to_mdfromarxiv2textimportarxiv_to_mdpdf_url="https://arxiv.org/pdf/1706.03762"output_dir="output_folder"arxiv_to_md(pdf_url,output_dir)arXiv APIimporturllib.requestaslibreqwithlibreq.urlopen('http://export.arxiv.org/api/query?search_query=all:electron&start=0&max_results=1')asurl:r=url.read()print(r)LicenseMITThe MIT License (MIT)

Copyright (c) 2023 Minwoo Park

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.Bugs and IssuesSincerely grateful for any reports on new features or bugs. Your valuable feedback on the code is highly appreciated.ContactsCore maintainer:Daniel Park, South KoreaE-mail:parkminwoo1991@gmail.comCopyright (c) 2023 MinWoo Park, South Korea
Package: arxivabscraper
arxivabscraperAn ArXiV scraper to retrieve abstracts from given categories and date range.InstallUsepip(orpip3for python3):$pipinstallarxivabscraperor download the source and usesetup.py:$pythonsetup.pyinstallor if you do not want to install the module, copyarxivabscraper.pyinto your working
directory.To update the module usingpip:pipinstallarxivabscraper--upgradeExamplesYou can directly usearxivabscraperin your scripts. Let's importarxivabscraperand create a scraper to fetch all preprints in condensed matter physics category
from 2 May 2018 until 2 June 2020 (for other categories, see below):importarxivabscraperscraper=arxivabscraper.Scraper(category='physics:cond-mat',date_from='2018-05-02',date_until='2020-06-02')Once we built an instance of the scraper, we can start the scraping:output=scraper.scrape()While scraper is running, it prints its status:fetching up to  1000 records...
fetching up to  2000 records...
Got 503. Retrying after 30 seconds.
fetching up to  3000 records...
fetching is complete.Finally you can save the output in your favorite format or readily convert it into a pandas dataframe:importpandasaspdcols=('categories','abstract')df=pd.DataFrame(output,columns=cols)CategoriesHere is a list of all categories available on ArXiv.CategoryCodeComputer SciencecsEconomicseconElectrical Engineering and Systems ScienceeessMathematicsmathPhysicsphysicsAstrophysicsphysics:astro-phCondensed Matterphysics:cond-matGeneral Relativity and Quantum Cosmologyphysics:gr-qcHigh Energy Physics - Experimentphysics:hep-exHigh Energy Physics - Latticephysics:hep-latHigh Energy Physics - Phenomenologyphysics:hep-phHigh Energy Physics - Theoryphysics:hep-thMathematical Physicsphysics:math-phNonlinear Sciencesphysics:nlinNuclear Experimentphysics:nucl-exNuclear Theoryphysics:nucl-thPhysics (Other)physics:physicsQuantum Physicsphysics:quant-phQuantitative Biologyq-bioQuantitative Financeq-finStatisticsstatContributingIdeas/bugs/comments? Please open an issue or submit a pull request on Github.LicenseThis project is licensed under the MIT License - see theLICENSEfile for details.AcknowledgmentsThis work is based on the arxivscraper from
Mahdi Sadjadi (2017). arxivscraper: Zenodo.http://doi.org/10.5281/zenodo.889853
Package: arxivAssistant
arxivAssistantA small package to access the daily astronomy articles from arXiv.org through the console. It relies ontextualfor an interactive exploration in an OS agnostic framework. This is very early work and I hope to add functionalities over the time!It can be installed with pip using the following commandpip install arxivAssistant --upgradeOnce installed, you can use it from the command line:arxiv-today
Package: arxiv-checker
arxiv-checker============A Python module to search arXiv.org. You can cross check a list of authors or keywords against either the most recent arXiv mailing or a given month/year.  Works with python 2.7 and python 3.To install::pip install arxiv-checkerTo check the most recent mailings of several arXiv subjects against a list of authors::import arxivcheckerpapers = arxivchecker.check_authors(['astro-ph', 'gr-qc', 'physics'], ['Smith, John', 'Doe, Jane'])When displayed, each paper lists the title, a clickable url link to the abstract, and the author list.To check against a long list of names, use a file::papers = arxivchecker.check_authors(['astro-ph', 'gr-qc', 'physics'], 'names.txt')If instead of names, you want to check each paper against a list of keywords in the title and abstract use::papers = arxivchecker.check_keywords('astro-ph', ['Planet Formation','Hot Jupiter'])You can also grab all of the papers first using the scrape_arxiv function::papers = arxivchecker.scrape_arxiv('astro-ph')results = arxivchecker.check_authors_from_papers(papers, 'Doe, Jane')Similarly, for checking keywords::papers = arxivchecker.scrape_arxiv('astro-ph')results = arxivchecker.check_keywords_from_papers(papers, ['GJ876','Gilese-876'])If you want to grab all of the papers from a given month you can supply the year and month arguments::papers = arxivchecker.scrape_arxiv('astro-ph',year=2016,month=6) # June 2016papers = arxivchecker.scrape_arxiv('astro-ph',year=2016,month='May') # May 2016Or grab all of the papers for a given year::papers = arxivchecker.scrape_arxiv('astro-ph',year=2016,month='all')Note however that this can take a while to complete (there could be more than 10,000 papers), and arXiv discourages against crawling through the website.Finally, to run straight from the command line::python -c "import arxivchecker; arxivchecker.check_authors('astro-ph', 'authors.txt') > results.txtTo email the results use the Unix mail command::mail -s "Arxiv Mailing" email@gmail.com < results.txtThe arxivchecker requires the request and bs4 modules.
Package: arxiv-cli
AboutA Python wrapper for the arXiv API.Install$pipinstallarxiv-cliUsageYou can usearXiv-CLIas a CLI to navigate arXiv or as a library to query
its API.CLICurrentlyarXiv-CLIimplements three subcommands:$arxivdownload[-t/--timeoutseconds]IDS$arxivfetch[-t/--timeoutseconds]IDS$arxivfind[-i/--ids,-t/--timeoutseconds]QUERYThe first two commands accept a list of arXiv ids and, respectively, download
the corresponding PDFs or display their metadata in JSON format.The third command runs a query against arXiv and prints the metadata in JSON
format of the records that match. Adding the-i/--idsflag will return only
their ids.Adding the-t/--timeoutsecondsoption will sleep for that amount of
seconds between successive requests to the arXiv API.Note that the previous commands can be chained, therefore running$arxivdownload$(arxivfind--idsQUERY)will download all papers that matchQUERY, while$arxivfetch$(arxivfind--idsQUERY)will fetch all their metadata.APIThe previous CLI is built on top of a Python library that can be used on its
own to query arXiv’s API. For example:>>>fromarxiv_cliimportClient>>>client=Client()>>>client.download([IDS])will achieve the same effect as$arxivdownloadIDSAuthorJacopo Notarstefano (@Jaconotar)LicenseMIT
Package: arxiv-decent-feeds
# arXiv decent feedsEver wanted to follow arXiv feeds on Slack, but try as you might slackbot wasunable to show arXiv updates?Wonder no more! arXiv decent feeds reads and converts arXiv RSS feeds to validRSS 2.0 feeds that slackbot will (hopefully) read!## Installation```pip install https://github.com/trovao/arxiv-decent-feeds/zipball/master```## Usage### Adding a feed```usage: arxivdf add [-h] url target_filepositional arguments:urltarget_fileoptional arguments:-h, --help   show this help message and exit```For example, to add the `cs.LG` (Computer Science Learning) feed, one could do:```arxivdf add http://export.arxiv.org/rss/cs.LG\?version\=2.0 cs.LG.rss```### Fetching updates```arxivdf update```### Generating feedsIf you added `cs.LG` as show above, after you can call the command `arxivdfgenerate` your feed will be saved to the file `cs.LG.rss` in the currentdirectory.### Periodically updating and generating feedsJust set up a cron job with to do that. What I'm currently using is, witha python virtualenv (the `source ~/venvs/feeds/bin/activate` part):```cron*/5 * * * * (source ~/venvs/feeds/bin/activate ; arxivdf update ; arxivdf generate)```
Package: arxiver
arxiver is an unoffical API for Cornell’s arxiv.org. This package allows access for search results and new publications in various topics.Contribute onGithubGetting startedInstallarxiverby running the following command:$ pip install arxiverAnd then in your Python file add:import arxiverThere are two main arxiver functions:get_new_publandsearch.get_new_publ(topic)Returns all new publications posted on arxiv.org given a certain topic in aPaperlist.Parameterstopic: string that identifies from which topic the new publications should be retrieved. List of valid topics:'astro-ph'      # Astrophysics
'cond-mat'      # Condensed Matter
'gr-qc'         # General Relativity and Quantum Cosmology
'hep-ex'        # High Energy Physics - Experiment
'hep-lat'       # High Energy Physics - Lattice
'hep-ph'        # High Energy Physics - Phenomenology
'hep-th'        # High Energy Physics - Theory
'math-ph'       # Mathematical Physics
'nlin'          # Nonlinear Sciences
'nucl-ex'       # Nuclear Experiment
'nucl-th'       # Nuclear Theory
'physics'       # Physics
'quant-ph'      # Quantum Physics

'math'          # Mathematics

'cs'            # Computer Science

'q-bio'         # Quantitative Biology
'q-fin'         # Quantitative Finance
'stat'          # StatisticsExampleReturn of list of papers from new publications in Computer Science and print out the link to each pdf:papers = arxiver.get_new_publ('cs')
for paper in papers:
    print paper.pdfsearch(query, max_results=30)Return list ofPaperobjects from arxiv given a search query and a maximum number of results.Parametersquery: string search query to pass to arxiv.orgmax_results: optional positive integer maxiumum number of results to be returned (default is 30)ExampleReturn a list of fifteen papers given ‘electron’ as a search term and print the abstract of each one:papers = arxiver.search('electron', max_results=15)
for paper in paper:
    print paper.abstractPaperPaperobjects have the following properties:Paper.titlethe title of the paperPaper.abstractthe abstract or short summaryPaper.arxiv_idunique string that identifies the arxiv paperPaper.pdflink to the pdf file of the whole paperPaper.pagelink to the page on arxiv.org that includes more informationPaper.authorslist of authors reprented by dictionaries with key'name'and'link'
Package: arxiv-miner
ArXiv-MinerArXiv Miner is a toolkit for mining research papers on CS ArXiv.What is ArXiv-Minerarxiv-mineris a quick handy library that helps powerSci-Genie. Sci-Genie is a search engine for quickly searching through full text of papers on CS ArXiv.arxiv-minerhelps extract and parse LaTeX documents from CS ArXiv. It also supports storage and search of those parsed documents usingElasticsearch. The library can be applicable for all other domains like Math, Physics, Biology etc.DocumentationAll documentation on how to install and usearxiv-mineris provided in thedocumentation websiteor inside thedocs folder. Contribution guidelines are also provided there.Why was ArXiv-Miner created ?ArXiv Miner was created for easily scraping, parsing and searching research content on ArXiv. This library was created after stitching together solutions from the code of various tools likearxiv-sanity,arxiv-vanity/engrafo,arxivscraper,tex2py,cso-classifierandaxcell. Parsed structure of the content can be useful in search or any scientific research mining/AI applications as a heuristic baseline.Core Components of ArXiv-MinerScrapingParsingIndexing/StorageFamily Of Projects With ArXiv-Minerarxiv-table-miner: Coming Soon.arxiv-table-ml-models: Coming Soon.semantic-scholar-data-pipeline:https://github.com/valayDave/semantic-scholar-data-pipelineDisclaimerThis project was developed like aCowboy coderover theCOVID-19 pandemic. Hence, thismay have bugs and not the most well optimized code. The primary reason for development was to aid CS and Machine Learning/AI research, but this tool can be extended to all 3M+ documents on ArXiv.Call For ContributorsAny help with contributions to improve the project or fix bugs are completely welcome. Please read the contribution guide in the documentation.Credits and AppreciationThis project like all others has been built on shoulders of giants. A big thanks to the creators of the following libraries/open source projects that aided the development ofarxiv-miner, and it's family of projects:arxiv-sanityarxiv-vanity/engrafoarxivscrapertex2pycso-classifieraxcellelasticsearchSemantic Scholar Open Research corpusmetaflowdocsifyLicenceMIT
Package: arxiv-post
arxiv-postTranslate and post arXiv articles to Slack and various appsInstallation$pipinstallarxiv-post
$playwrightinstallchromiumUsageCommand line interfacearxiv-postis available after installation, with which you can translate and post arXiv articles to various apps.
Note that onlyslackapp is currently available.
You need tocreate a custom Slack app to get an URL of incoming webhook.$arxiv-postslack--keywordsdeshima\--categoriesastro-ph.IM\--target_langja\--slack_webhook_url<SlackwebhookURL>The posted article looks like this.For detailed information, see the built-in help by the following command.$arxiv-postslack--helpExampleIt would be nice to regularly run the command by some automation tools such as GitHub Actions.
Here is a live example where daily arXiv articles inastro-ph.GA,astro-ph.IM, andastro-ph.HEare posted to different channels of a Slack workspace.a-lab-nagoya/astro-ph-slack: Translate and post arXiv articles to SlackReferencesfkubota/Carrier-Owl: arxiv--> DeepL --> Slack: The arxiv-post package is highly inspired by their work.
Package: arxivpy
Find all papers from arXiv based on some query ( + option for downloading them )Installation>>> pip install arxivpyDocumentationOfficial documentation can be foundhereExample usage>>> from arxivpy.client import ArXivPyClient
>>>
>>> client = ArXivPyClient()
>>>
>>> quantum = client.simple(query='quantum computing', page=2, download=True,
>>>                         path='path', downloadonly=10)
>>>
>>>
>>> algorithms = client.advanced(term='algorithms',
>>>                              classification='computer_science', size=200)
>>>
>>>
>>>
>>> alg_ds = client.advanced(terms=[{term: 'algorithms', field: 'title'},
>>>                                 {term: 'data structures', field: 'title'}],
>>>                      classification='computer_science', size=200)
>>>LicenceMIT
Package: arxiv-rename
arXiv RenamerThis package has a utility calledarxiv-renamerthat can be used to renamepdffiles downloaded fromarXiv.
It creates a slug of the form<id>-<authors>-<title>and renames the original file to that slug.Another utility calledarxiv-watchercan be used to watch for new (or moved) files in a directory and rename them.Installationpipinstallarxiv_renameUsageusage:arxiv_rename[-h][-w]PATH

RenamePDFsdownloadedfromarXiv.org

positionalarguments:
PATHpathtoarXivPDForafolderofarXivPDFs

optionalarguments:
-h,--helpshowthishelpmessageandexit-w,--watchwatchfornewfilesinthefolderspecifiedbyPATH
Package: arxivs
arxivsGet arXiv AI paper.InstallpipinstallarxivsUsageimportarxivspaper_list=arxivs.get_paper(tags=["cs.CL"],days=1)print(paper_list[0])Tag Listcs.AI:ArtificialIntelligence
cs.CL:ComputationandLanguage
cs.CV:ComputerVisionandPatternRecognition
cs.FL:FormalLanguagesandAutomataTheory
cs.IR:InformationRetrieval
cs.LG:Learning
cs.MA:MultiagentSystems
cs.NE:NeuralandEvolutionaryComputing
stat.ML:MachineLearningLICENSE
Package: arxivterminal
Arxiv TerminalArxiv Terminal is a command-line interface (CLI) tool for fetching, searching, and displaying papers from thearXivpreprint repository. The tool allows you to fetch papers from specified categories, search the fetched papers, and display their statistics.FeaturesFetch paper abstracts from specified categories and save them in a local sqllite database.Show fetched papers and interatively open for more detailed abstractsSearch fetched papers based on a query (Currently supports pattern + LSA semantic search)Download papers locally as PDFContributorsA special call out to ChatGPT (v4) which helped write and modify various code and documentation in this repository.InstallationpipinstallarxivterminalFor local builds, you should have Poetry installed:User Guide. After
installation you may clone and build this repo:poetryinstall
poetryshell
arxiv<command># Build the wheelspoetrybuildUsageThe CLI is invoked using thearxivcommand, followed by one of the available commands:arxiv fetch [--num-days] [--categories]: Fetch papers from the specified categories and store them in the database.arxiv delete_all: Delete all papers from the database.arxiv show [--days-ago]: Show papers fetched from the specified number of days ago.arxiv stats: Show statistics of the papers stored in the database.arxiv search <query>: Search papers in the database based on a query.ExamplesFetch papers from the "cs.AI" and "cs.CL" categories from the last 7 days:arxivfetch--num-days7--categoriescs.AI,cs.CLDelete all papers from database:arxivdelete_allShow papers fetched in the last 7 daysarxivshow--days-ago7# On this screen the user can select a paper for more# details by typing a line number. Additional options# allow for searching & downloadingDisplay statistics of the papers stored in the database:arxivstatsShow papers containing the phrase "deep learning":arxivsearch"deep learning"Show papers containing the phrase "deep learning" using LSA matching:arxivsearch-e"deep learning"LSA Search ModelNote: This approach is likely to be replaced in the future by more robust methodologyThe LSA search model is largely adapted from the implementation featured in the scikit-learnUser Guideexample.
When used, the model is trained over the entire corpus of abstracts present in the user's local database. The model
is persisted in the app cache folder and automatically reloaded on subsequent runs. During a search query, all abstracts
from the database are encoded as n-dimensional vectors using the trained LSA model. The search query is also represented
as a vector, and a cosine similarity is performed to find the top ranking items.You may want to force a refresh of the underlying model after loading new papers. This can be done by using the-fflag when performing a search:arxivsearch-e-f"deep learning"