Package: yamconv
yamconvyamconvconverts a machine learning dataset from one format to another format.Installationyamconvis published onPyPI. You can installyamconvusing pip as follows:pipinstallyamconvAlternatively, you can install it from the source code by runningpipin the project directory wheresetup.pyis located:pipinstall.Usageyamconv.py-cconverter-iinput_file-ooutput_file-ssettings-v-c: converter name-i: input file path-o: output file path-s: converter settings in JSON-v: verbose, to display the processing progress and informationSupported convertersThe following are the supported converters:mlt.sqlite2fasttext: SQLite database file to fastText text filemlt.sqlite2csv: SQLite database file to CSV text filemlt.fasttext2sqlite: fastText text file to SQLite database filemlt.csv2sqlite: CSV text file to SQLite database filemlt.csv2fasttext: CSV text file to fastText text filemlt.sqlite2sqlite: SQLite database file to SQLite database file (with normalization)mlt.fasttext2fasttext: fastText text file to fastText text file (with normalization)mlt.csv2csv: CSV text file to CSV text file (with normalization)SettingsSettings for converters are given in the-soption as a JSON string, e.g.,'{"cache_labels": true}'.SettingValuesDescriptionApplicable convertersnormalize_labelstrue(default),falseWhennormalize_labelsistrue, all labels are normalized. That is, all symbols are removed; all alphabets are converted to lower case.Anyword_seqtrue,false(default)Whenword_seqistrue, each text is normalized into a sequence of lower-case words. That is, all symbols are removed, all alphabets are converted to lower case; and all unicode word characters (e.g., Chinese characters) are delimited by a space.Anycache_labelstrue,false(default)Whencache_labelsistrue, the normalized labels are cached in memory. It can be set tofalseif there is insufficient memory to cache a huge number of different labels in the dataset.AnySupported dataset formatsMulti-label text classificaitonSQLite databaseASQLitedatabase is used to store the classifications of texts.
The database schema is as follows:CREATETABLEIFNOTEXISTStexts(idTEXTNOTNULLPRIMARYKEY,textTEXTNOTNULL);CREATETABLEIFNOTEXISTSlabels(labelTEXTNOTNULL,text_idtextNOTNULL,FOREIGNKEY(text_id)REFERENCEStexts(id));CREATEINDEXIFNOTEXISTSlabel_indexONlabels(label);CREATEINDEXIFNOTEXISTStext_id_indexONlabels(text_id);Thetextstable contains the text contents in thetextfield,
and each row is uniquely identified by theidfield.
Thelabelstable contains the labels in thelabelfield.
Each row has atext_idforeign key that links the label to the text in thetextstable,
where the text is classified with the label.
In other words, each row intextsis associated with zero or more rows inlabels.fastText text fileThefastTextformat is a text file that contains a series of lines.
Each line represents a text classified by multiple labels.
A line starts with multiple labels, followed by the text content.
Each label is marked with the__label__prefix and the labels are separated by a space.
The following is a fragment of an example fastText dataset file:__label__food __label__region Many people love having dim sum in Hong Kong restaurants.
__label__region __label__plant __label__business The Netherlands is the major supplier to the European floral market.CSV text fileThe dataset is in form of a CSV (Common Separated Values) file. The first row is the header. Each of the second row and the following rows stores a single record. The CSV file can be in either of one of the following formats.Format 1Suppose the format of the header row is like the follwoing:"id", "text", "region", "business", "food", "plant"That is:Cell1:idCell2: any arbitary valueCellnwheren >= 3: the name of labeln, e.g.,region,business,food,plant.Each record row looks like:"10", "Many people love having dim sum in Hong Kong restaurants.", 1, 0, 1, 0That is:Cell1: theidstringCell2: the text contentCellnwheren >= 3:1or0representing whether the text is classified with labelnor not respectively.Format 2Suppose the format of the header row is like the follwoing:"text", "region", "business", "food", "plant"That is:Cell1: any arbitary valueCellnwheren >= 2: the name of labeln, e.g.,region,business,food,plant.Each record row looks like:"Many people love having dim sum in Hong Kong restaurants.", 1, 0, 1, 0That is:Cell1: the text contentCellnwheren >= 2:1or0representing whether the text is classified with labelnor not respectively.Profesional servicesIf you need any supporting resources or consultancy services from YAM AI Machinery, please find us at:https://www.yam.aihttps://twitter.com/theYAMaihttps://www.linkedin.com/company/yamaihttps://www.facebook.com/theYAMaihttps://github.com/yam-aihttps://hub.docker.com/u/yamai