Package: readnext
Read NextReadNext: A Personal Papers RecommenderEvery day, approximately 500 new papers are published in thecscategory on arXiv, with tens more incs.AIalone. Amidst the recent
craze around Generative AI, I found it increasingly challenging to keep
up with the rapid influx of papers. Distilling the ones that were most
relevant to my work and my employer’s interests became a daunting task.ReadNext was born out of these pressing needs:The necessity for a command-line tool, one that could be executed
directly or scheduled as a cron job.The requirement to access the latest papers fromarXiv.The integration withZotero, an excellent
tool for managing academic papers.The ability to propose a selection ofxpotentially relevant papers
based on my current research focus.The proposed papers should be accessible from the command line.The proposed papers should also be easily uploaded to Zotero.The key focus is to recommend papers that align with my evolving
interests and research objectives, which may change on a daily basis and
need to be continuously accounted for.InstallYou can easily install the ReadNext command line tool usingpip:pipinstallreadnextRequirementsReadNext relies on one fundamental external services to create the
propose user workflow:Zotero: Zotero serves as the primary papers management tool, playing
a pivotal role in ReadNext’s workflow. To configure ReadNext on your
local computer, you have to create a Zotero account. If you do not
already have one, you will have to create one for yourself, please
refer to the section below.By integrating Zotero, ReadNext helps in discovering papers that align
with your research interests and focus.The second piece of the command line tool is the embedding system. That
system is used to create embeddings that are used to recommend papers to
users according to their current research focus. By default, ReadNext is
using theBAAI/bge-base-enmodel from Hugging
Face. Optionally, you can use
the Cohere embedding service instead. The processing is a little bit
faster depending on your local desktop, but it requires and additional
dependency.Also note that the performance between the two systems are comparable.
In my experience, about 80% of the propositions are the same, and the
remaining 20% that are different yeld no major difference in accuracy.
However, I do prefer theBAAI/bge-base-enpropositions a little
better.Zotero AccountIf you do not have a Zotero account yet, you cancreate one here for
free. This will give you a basic
account with200MBof space, sufficient to get started. Afterwards,
you can install the desktop application, mobile apps, and necessary
browser plugins to fully integrate Zotero into your digital environment.Take the time to refer totheir extensive online
documentationto get to know its full
potential.Cohere Account (Optional)For Cohere, you will have tocreate an account and login on their
Dashboard. The services
are completely free for the volume and size of request required by
ReadNext.ConfigureReadNext currently needs to be configured using the environment
variables of your terminal session. The following configuration options
currently needs to be configured:OptionDescriptionCOHERE_API_KEYCohere API Key ascreated in their Dashboard hereZOTERO_LIBRARY_IDYour personal library ID as defined in Zotero’s backend. ThisID will appears hereasYour userID for use in API calls is 750ZOTERO_LIBRARY_TYPEType of library:userorgroupZOTERO_API_KEYYou Zotero API Key,it needs to be created and managed here.CHROMA_DB_PATHThis is the local path where you want the embedding database management system to save its indexes (ex:/Users/me/.readnext/chroma_db/)EMBEDDING_SYSTEMThis is the embedding system you want to use. One of:BAAI/bge-base-en(local) orcohere.MODELS_PATHThis is the local path where you want the models files to be saved on your local file system (ex:/Users/me/.readnext/models/)DOCS_PATHThis is the local path where you want the PDF files of the papers from arXiv to be saved locally (ex:/Users/me/.readnext/docs/)RECOMMENDATIONS_PATHThis is the local path where you want the recommended papers to be saved locally (ex:/Users/me/.readnext/recommendations/)Setup Environment VariablesFor WindowsOpen the Command Prompt or PowerShell.Use thesetxcommand to create a new environment variable
permanently. For example, to set theZOTERO_API_KEY, you can use
the following command:setxZOTERO_API_KEY"your_zotero_api_key_here"The changes will take effect after you open a new command prompt or
restart your computer.MacOS and Linux:Open the Terminal.Use theexportcommand to set the environment variable for the
current session. For example, to set theZOTERO_API_KEY, use the
following command:exportZOTERO_API_KEY="your_zotero_api_key_here"This will set theZOTERO_API_KEYvariable for the current session
only.To make the variable permanent, add theexportcommand to your shell’s
configuration file. For example, if you’re using Bash, add the line to
the~/.bashrcor~/.bash_profilefile. If you’re using Zsh, add it
to the~/.zshrcfile. You can do this with a text editor or by using
the following command:echo'export ZOTERO_API_KEY="your_zotero_api_key_here"'>>~/.bashrcReplace~/.bashrcwith the appropriate file name if you’re using a
different shell configuration.After saving the environment variables, they will be available in your
command line sessions, and any application that relies on them, such as
ReadNext, will be able to access the configured values. Remember to
restart your command line or open a new session after making changes to
ensure the environment variables take effect.You can verify that the environment variables are set by running theenvcommand in your terminal session.Here is what the full export looks like:exportCOHERE_API_KEY=""exportZOTERO_LIBRARY_ID=""exportZOTERO_API_KEY=""exportZOTERO_LIBRARY_TYPE="user"exportEMBEDDING_SYSTEM="BAAI/bge-base-en"exportCHROMA_DB_PATH="/Users/[MY-USER/.readnext/chroma_db/"exportMODELS_PATH="/Users/[MY-USER]/.readnext/models/"exportDOCS_PATH="/Users/[MY-USER]/.readnext/docs/"exportRECOMMENDATIONS_PATH="/Users/[MY-USER]/.readnext/recommendations/"How it works?ReadNext is a command line tool that can be used to generate
personalized paper recommendations based on your research interests. It
is designed to be used as a daily routine to help you discover new
papers that are relevant to your research focus.The tool is designed to be used in conjunction withZotero, a free and open-source reference
management software to manage your research library. ReadNext will use
your Zotero library to identify your research interests and focus, and
will propose papers that are relevant to your research focus.ReadNext is designed to be used as a daily routine. It will propose a
list of papers that are relevant to your research focus, and will save
them in a dedicated collection in your Zotero library. You can then
review the proposed papers and decide which ones you want to read. Once
you have read a paper, you can move it to another collection in your
Zotero library, and ReadNext will learn from your feedback to improve
the quality of the proposed papers.ReadNext is designed to be used with with an embedding system. It uses
the system to generate embeddings for the papers in your Zotero library,
and will use these embeddings to identify papers that are similar to
your research focus. Two embedding systems are currently support: a
local one using theBAAI/bge-base-enmodel from Hugging Face, and a
remote one using the Cohere embedding service.ReadNext is designed to be used witharXiv, a free
service that provides access to scientific papers in the fields of
mathematics, physics, astronomy, computer science, quantitative biology,
statistics, and quantitative finance. ReadNext will use arXiv to
identify the latest papers in your research focus, and will propose them
to you as part of your daily routine.The designed userflow is the following:As a Zotero user, I will create one or multiple “Focus” collections
in my Zotero library. Those are the collections where I will add the
papers that are the most interesting to my current research. It is
expected that the content of those collections will change over time
as my research focus and interests evolves.On a daily basis, I will runreadnextin my terminal, or I will
create a cron job to run it automatically for me.ReadNext will fetch the latest papers from arXivReadNext will identify the papers that are relevant to your
research focus, as defined in ZoteroReadNext will propose the relevant papers to me and add them to
Zotero in a dedicated collection where proposed papers are savedI will go in Zotero, start to read the proposed papers, and if any
are of a particular interest I will add them to one of the “Focus”
collectionsReadNext will learn from your feedback to improve the quality of the
proposed papersNow, let’s see how to actually do this.UsageHelpAny time, you can get contextual help for any command like this:readnext--helpreadnextpersonalized-papers--helpThose commands will tell you which arguments and options are available
for each command.arXiv categories and subcategoriesYou can get the full list of arXiv categories and subcategories by
running the following command:readnextarxiv-top-categoriesreadnextarxiv-sub-categoriesThose are the categories where you can get specific new papers from
arXiv.Getting new papers proposalsTo get new papers proposals, you have to run thepersonalized-paperscommand. That command requires two arguments:category: the arXiv top, or sub, category from which you want to get
new papers proposalszotero_collection: the name of the Zotero collection where your
papers of interest are stored in Zotero. This is what we refer to as
the “Focus” collection above. The name of the collection is case
sensitive and should be exactly as written in Zotero.Then you also have three options available:--proposals-collection: which tells ReadNext that you want to save
the proposed papers in Zotero, in the Zotero Collection specified by
the argument. If you don’t use this option, ReadNext will only print
the proposed papers in the terminal, but will not save them in Zotero.
The default behaviour is that you don’t save them in Zotero.--with-artifacts/-a: which tells ReadNext that you want to save
the artifacts (PDF file of the papers and their summarization) into
Zotero. This is the recommended workflow, but it requires a lot more
space in your Zotero account. If you want to do this, you will most
likely need to subscribe to one of their paid option.--nb-proposals: which tells ReadNext how many papers you want to be
proposed. The default value is 10.The following command will propose 3 papers from thecs.AIcaterory,
based on theReadnext-Focus-LLMcollection in my Zotero library, save
them in Zotero in theReadnext-Propositions-LLMwith all related
artifacts:readnextpersonalized-paperscs.AIReadnext-Focus-LLM--proposals-collection=Readnext-Propositions-LLM--with-artifacts--nb-proposals=3As you can see, you can easily create a series of topics you want papers
proposals around, where each of the topic is defined by a series of
specific papers that you read and found important for your research.Here is what it looks like in the terminal:Here is what it looks like in Zotero:Future WorkHere is a list of future work that could be done to improve ReadNext
after the initial release:Adding an Abstraction Layer for Multiple Embedding Services:
Currently, ReadNext utilizes Cohere for embeddings in its initial
version. LangChain could serve as a potential abstraction layer to
support multiple different embedding services.Expanding Paper Sources with an Abstraction Layer: ReadNext aims
to integrate additional paper sources beyond just arXiv. This will be
facilitated by implementing an abstraction layer for seamless
integration.Enhancing Test Coverage: To improve testing, we will go beyond
testing utility functions and incorporate mocks for external services,
ensuring comprehensive test coverage.Interactive Configuration via Command Line Tool: We plan to
augment the command line tool’s functionality, allowing users to
configure it directly from the command line using appropriate prompts
and interactions.Refining Paper Selection Process: Currently, every time readnext
is executed, it retrieves today’s latest papers from arXiv, identifies
relevant papers based on current interests, and matches them against
the personal research focus. To further enhance this functionality, we
may introduce additional capabilities, such as restricting proposed
papers to today’s papers only.ContributionsWe welcome contributions to ReadNext! If you’d like to contribute,
please follow these steps:Fork the repository on GitHub.Create a new branch with a descriptive name:git checkout -b feature/your-feature-nameMake your changes and commit them:git commit -m "Add feature: your feature name"Push your changes to your fork:git push origin feature/your-feature-nameSubmit a pull request to themainbranch of the original
repository.