Package: xlingual-papers-recommender
What is it?Component of a papers recommender system in a cross-lingual and multidisciplinary scope.Result of the Coursework of MBA in Data Science and Analytics - USP / ESALQ - 2020-2022.Designed to be customizable in many ways:sentence-transformer modelthe maximum number of candidate articles for the evaluation of semantic similarityaccepts any type of document that has bibliographic referencesDependencessentence-transformercelerymongoengineModelThe algorithm adopted is a combination of recommender systems graph based and content based filtering with semantic similarityThe identification of the relationship between scientific articles is made during the document's entry into the system through the common bibliographic references. Subsequently, the documents are ranked by semantic similarity and recorded in a database.The recommendation system works in two steps: creating links between articles via common citations and assigning a similarity coefficient for a selection of these linked articles.The system itself does not establish which articles should be recommended.The recommendation system client defines which articles to present as a recommendation depending on the criticality of the use case.Installationpip install -U xlingual_papers_recommenderConfigurationsexport DATABASE_CONNECT_URL=mongodb://my_user:my_password@127.0.0.1:27017/my_dbexport CELERY_BROKER_URL="amqp://guest@0.0.0.0:5672//"export CELERY_RESULT_BACKEND_URL="rpc://"export MODELS_PATH=sentence_transformers_modelsexport DEFAULT_MODEL=paraphrase-xlm-r-multilingual-v1CeleryStart servicecelery -A xlingual_papers_recommender.core.tasks worker -l info -Q default,low_priority,high_priority --pool=solo --autoscale 8,4 --loglevel=DEBUGClean queuecelery worker -Q low_priority,default,high_priority --purgeUsageRegister new paperxlingual_papers_recommender receive_paper [--skip_update SKIP_UPDATE] source_file_path log_file_pathpositional arguments:
source_file_path      /path/document.json
log_file_path         /path/registered.jsonloptional arguments:
-h, --help            show this help message and exit
--skip_update SKIP_UPDATE
if it is already registered, skip_update do not updateExamples of source_file_path:docs
└── examples
    ├── document1.json
    ├── document2.json
    ├── document3.json
    ├── document4.json
    ├── document5.json
    ├── document51.json
    ├── document6.json
    ├── document6_2.json
    ├── document7.json
    └── document7_2.jsonReferences attributes:pub_yearvolnumsupplpagesurnameorganization_authordoijournalpaper_titlesourceissnthesis_datethesis_locthesis_countrythesis_degreethesis_orgconf_dateconf_locconf_countryconf_nameconf_orgpublisher_locpublisher_countrypublisher_nameeditionsource_person_author_surnamesource_organization_authorGet paper recommendationsusage: xlingual_papers_recommender get_connected_papers [-h] [--min_score MIN_SCORE] pidpositional arguments:pid                   pidoptional arguments:-h, --help            show this help message and exit--min_score MIN_SCOREmin_scoreLoad papers data from datasetsRegister partsusage: xlingual_papers_recommender_ds_loader register_paper_part [-h] [--skip_update SKIP_UPDATE] [--pids_selection_file_path PIDS_SELECTION_FILE_PATH]{abstracts,references,keywords,paper_titles,articles} input_csv_file_path output_file_pathpositional arguments:{abstracts,references,keywords,paper_titles,articles}part_nameinput_csv_file_path   CSV file with papers part dataoutput_file_path      jsonl output file pathoptional arguments:-h, --help            show this help message and exit--skip_update SKIP_UPDATETrue to skip if paper is already registered--pids_selection_file_path PIDS_SELECTION_FILE_PATHSelected papers ID file path (CSV file path which has "pid" column)Register articlesExample:xlingual_papers_recommender_ds_loader register_paper_part articles articles.csv articles.jsonlRequired columnspidmain_langurisubject_areaspub_yeardoi (optional)network_collection (optional)Register abstractsExample:xlingual_papers_recommender_ds_loader register_paper_part abstracts /inputs/abstracts.csv /outputs/abstracts.jsonlColumnspidlangoriginaltext (padronizado)Same forpaper_titlesandkeywordsdatasets.Register referencesExample:xlingual_papers_recommender_ds_loader register_paper_part references /inputs/references.csv /outputs/references.jsonlColumnspub_yearvolnumsupplpagesurnameorganization_authordoijournalpaper_titlesourceissnthesis_datethesis_locthesis_countrythesis_degreethesis_orgconf_dateconf_locconf_countryconf_nameconf_orgpublisher_locpublisher_countrypublisher_nameeditionsource_person_author_surnamesource_organization_authorMerge papers partsusage: xlingual_papers_recommender_ds_loader merge_parts [-h] [--split_into_n_papers SPLIT_INTO_N_PAPERS] [--create_paper CREATE_PAPER]
                                                         input_csv_file_path output_file_path

positional arguments:
  input_csv_file_path   Selected papers ID file path (CSV file path which has "pid" column)
  output_file_path      jsonl output file path

optional arguments:
  -h, --help            show this help message and exit
  --split_into_n_papers SPLIT_INTO_N_PAPERS
                        True to create one register for each abstract
  --create_paper CREATE_PAPER
                        True to register papersExample:xlingual_papers_recommender_ds_loader merge_parts pids.csv output.jsonlRegister papers from loaded datasetsusage: xlingual_papers_recommender_ds_loader register_paper [-h] [--skip_update SKIP_UPDATE] input_csv_file_path output_file_path

positional arguments:
  input_csv_file_path   Selected papers ID file path (CSV file path which has "pid" column)
  output_file_path      jsonl output file path

optional arguments:
  -h, --help            show this help message and exit
  --skip_update SKIP_UPDATE
                        True to skip if paper is already registeredExample:xlingual_papers_recommender_ds_loader register_paper pids.csv output.jsonlGenerate reports from papers, sources and connectionsusage: xlingual_papers_recommender_reports all [-h] reports_path

positional arguments:
  reports_path  /path

optional arguments:
  -h, --help    show this help message and exitExample:xlingual_papers_recommender_reports all /reports