Package: tatoebatools
tatoebatoolstatoebatoolshelps you to integrateTatoebainto your app more quickly by allowing you to easily download and parseTatoeba weekly exports.InstallationThis library supports Python 3.8+.pipinstalltatoebatoolsBasic UsageUse the high-levelParallelCorpusclass to automatically download and iterate through all sentence/translation pairs from a source language to a target language.>>>fromtatoebatoolsimportParallelCorpus>>>forsentence,translationinParallelCorpus("cmn","eng"):print((sentence.text,translation.text))...('那里有八块小圆石。','There were eight pebbles there.')('这个椅子坐着不舒服。','This chair is uncomfortable.')('我会在这里等着到他回来的。','Until he comes back, I will wait here.')Advanced UsageThe Tatoeba data files are handled by thetatoebaobject.fromtatoebatoolsimporttatoebaBy default, the fetched Tatoeba data files are stored inside thetatoebatoolspackage. But you can download them to another location.tatoeba.dir="/path/to/my/tatoeba/dir"Use theall_tablesattribute to list the Tatoeba data tables you can have access
to.>>>tatoeba.all_tables['jpn_indices','links',...,'user_languages','user_lists']Each table has its own set of attributes:TableAttributessentences_detailedsentence_id, lang, text, username, date_added, date_last_modifiedsentences_basesentence_id, base_of_the_sentencesentences_CC0sentence_id, lang, text, date_last_modifiedlinkssentence_id, translation_idtagssentence_id, tag_namesentences_in_listslist_id, sentence_idjpn_indicessentence_id, meaning_id, textsentences_with_audiosentence_id, audio_id, username, license, attribution_urluser_languageslang, skill_level, username, detailstranscriptionssentence_id, lang, script_name, username, transcriptionuser_listslist_id, username, date_created, date_last_modified, list_name, editable_byFind out more about the Tatoeba data files and their fieldshere.Most Tatoeba languages are identified by theirIS0 639-3 codes. The asterisk character '*' designates all languages supported by Tatoeba. Call theall_languagesattribute to list the languages supported by Tatoeba.>>>tatoeba.all_languages['abk','acm',...,'zul','zza']Reading Tatoeba data with iteratorsTo read a table, just call its iterator. The downloading of the data file will be automatically handled in the background.Set thescopeargument to 'added' to only read rows that did not exist in the previous local version of an updated file. Set it to 'removed' to iterate over the rows that don't exist anymore.# list all sentences in Englishenglish_texts=[s.textforsintatoeba.sentences_detailed("eng")]# list all German sentences that were added by the latest local updatenew_german_texts=[s.textforsintatoeba.sentences_detailed("deu",scope="added")]# list all links between French and Italian sentencesfrench_italian_links=[(lk.sentence_id,lk.translation_id)forlkintatoeba.links("fra","ita")]# list all French native speakersnative_french=[x.usernameforxintatoeba.user_languages("fra")ifx.skill_level==5]# map German sentences to their audiosgerman_audios={}foraudiointatoeba.sentences_with_audio("deu"):german_audios.setdefault(audio.sentence_id,[]).append(audio.audio_id)Extracting Tatoeba data as dataframeSincetatoebatoolsrelies heavily on thepandaslibrary, it is possible to load any supported table into memory as a dataframe.# get the dataframe of the English sentences tableenglish_sentences_dataframe=tatoeba.get("sentences_detailed",["eng"])# get the dataframe of all links for which French is the source languagefrench_links_dataframe=tatoeba.get("links",["fra","*"])Ingesting Tatoeba data into a databaseThetatoebatoolslibrary includesSQLAlchemymodels that help you to ingest Tatoeba data in the database of you choice.In the example below, all sentence and user data is loaded into a local SQLite database, and then an index is added to speed up database queries.fromsqlalchemyimportcreate_engine,Indexfromtatoebatoolsimporttatoebafromtatoebatools.modelsimportBase,SentenceDetailedengine=create_engine("sqlite:///./tatoeba.sqlite")table_names=["sentences_detailed","user_languages"]# create the tables in the databasetables=[Base.metadata.tables[table_name]fortable_nameintable_names]Base.metadata.create_all(bind=engine,tables=tables)# insert data into the tablesfortable_nameintable_names:withtatoeba.get(table_name,["*"],chunksize=100000)asreader:forchunkinreader:chunk.to_sql(table_name,con=engine,index=False,if_exists="append")# add an index on the 'username' column of the 'sentences_detailed' tableix=Index('ix_sentences_detailed_username',SentenceDetailed.username)ix.create(engine)