Package: tmxutil
tmxutiltmxutil.py allows you to add domain groups to your Europat tmx files, or filter on them.Installation & RequirementsTo install tmxutil.py, just download it fromGithuband place it somewhere where you can reach it from the command line. Besides Python 3.5 or newer, it has no external dependencies.ExamplesExample tmx file:DE-EN-2001-Abstract.tmx.gz, ipc domain group file:ipc-groups.tabThe provided IPC grouping has the following high-level categories:GroupDescriptionIGeneral / DefaultIIComputing, Science and Tech (Science, photography, optics, cryptography, communications)IIIBiotechnology and Chemical (food, biotech, nanotech, chemistry)IVEngineering and Manufacturing (Engines, nuclear physica, agriculture, forestry, aviation)VDaily life (Household, music, arts, clothing, jewlery, sports and decorating)Filtering by IPC code: Filter out only sentence pairs that come from patents with a certain IPC codes.gzip -cd DE-EN-2001-Abstract.tmx.gz \
| ./tmxutil.py -o tmx --with-ipc D06M15/59 D06P005/02 \
> selection.tmxExport selection as tab-separated sentence pairs: By changing the output format of tmxutil you can get the sentence pairs as plain text separated by tabs.This option can be combined with data augmentation and filter options, although only the first source document per sentence pair is exported. You'll also have to tell it in what order you want the languages to be exported.gzip -cd DE-EN-2001-Abstract.tmx.gz \
| ./tmxutil.py \
    -o tab \
    --output-languages en de \
    --with-ipc D06M15/59 \
> selection-en-de.tsvAdding ipc groups to tmx file: To be able to make more coarse-grained selections you can add ipc groups (c.f. domains) to the sentence pairs, based on the IPC codes already in the tmx file. You can then use those ipc groups to make a selection using--with-ipc-group, which works just like--with-ipc.The ipc-groups.tab file used here should have a IPC code prefix and a group name on each line, separated by a tab, as the first two columns. You can get the ipc-groups.tab file fromthe project's releases page.gzip -cd DE-EN-2001-Abstract.tmx.gz \
| ./tmxutil.py \
	-o tmx \
	--ipc-group ipc-groups.tab \
| gzip > DE-EN-2001-Abstract-with-groups.tmx.gzOnly the tmx output format will maintain the ipc-group metadata by adding ipc-group properties. Other output formats won't maintain it, but you can still use--with-ipc-groupdirectly to make a selection.Converting tsv to tmx: tmxutil can also be used to generate tmx files from sentence pairs. The input format is the same as thetaboutput format, that issource1 \t source2 \t sentence1 \t sentence2.To also add the IPC codes from metadata, use the--ipcoption. The format of this file should bel1_id \t _ \t _ \t _ \t l1_lang \t l1_ipcs \t l2_id \t _ \t _ \t _ \t l2_lang \t l2_ipcswhereidis the document identifier, andl1_ipcis a comma-separated list of all ipc codes for this document.cat DE-EN-2001-Abstract-aligned.tsv \
| ./tab2tmx.py \
    -o tmx \
    -l de en \
    -d \
    --ipc DE-EN-2001-Metadata.tab \
| gzip -9c > DE-EN-2001-Abstract.tmx.gzParameters-i tmx|tab, --input-format tmx|tabinput format, if not given will be auto-detected. Possible values:tmx,tab.In case oftabyou'll have to specify which languages are in there using--languages l1 l2.-o tmx|tab|txt, --output-format tmx|tab|txtoutput format, eithertmx,tabortxt.In case oftabyou'll have to specify the languages, e.g.--output-languages l1 l2.When usingtxt, you'll have to select which language you want the plain text for, i.e.--output-languages en.-l L1 L2, --input-languages L1 L2. Languages & order of them in the input file. Only necessary when readingtabfiles.--ouput-languages L1 [L2]language or order of languages in the output file. Not used iftmxis the output.-d, --deduplicategroups sentence pairs with the same text or hash together.--drop PROP [PROP ...]drop properties from the sentence pairs while writing output.--renumber-outputcauses all translation unit ids to be reset. Enabled by default when multiple input files are given.--ipc FILEadds IPC metadata to each sentence pair.--with PROP=VALUE [PROP=VALUE ...]filters sentence pairs on their text or properties. Supported operators are=,>,<,>=,<=and=~for regular expressions. Use multiplePROP=VALUEpairs in a--withoption to combine the conditions (i.e. AND). Or use multiple--withoptions for separate conditions (i.e. OR).--without PROP=VALUE [PROP=VALUE ...]same as--with, but negated, for excluding instead of including sentence pairs.--verboseenabled progress updates.