Package: text-selection
text-selectionCommand-line interface (CLI) to select lines of a text file.Featuresdatasetcreate: create a dataset based on a text fileexport-statistics: exporting statistics to a CSVsubsetsadd: add subsetsremove: remove subsetsrename: rename subsetselect-all: select all linesselect-fifo: select lines FIFO-styleselect-greedily: select lines greedily regarding unitsselect-greedily-ep: select lines greedily regarding units (epoch-based)select-uniformly: select lines with units uniformly distributedselect-randomly: select lines randomlyfilter-duplicates: filter duplicate linesfilter-by-regex: filter lines by regexfilter-by-text: filter lines by textfilter-by-weight: filter lines by weightfilter-by-vocabulary: filter lines by unit vocabularyfilter-by-count: filter lines by global unit frequenciesfilter-by-unit-freq: filter lines by unit frequencies per linefilter-by-line-nr: filter lines by line numbersort-by-line-nr: sort lines by line numbersort-by-text: sort lines by textsort-by-weight: sort lines by weightssort-by-shuffle: shuffle linesreverse: reverse linesexport: export linesweightscreate-from-file: create weights from filecreate-uniform: create uniform weightscreate-from-count: create weights from unit countdivide: divide weightsRoadmapadd testsrefactoringoutsourcing greedy- and KLD-iteratorInstallationpipinstalltext-selection--userUsageusage: text-selection-cli [-h] [-v] {dataset,subsets,weights} ...

CLI to select lines of a text file.

positional arguments:
  {dataset,subsets,weights}  description
    dataset                  dataset commands
    subsets                  subsets commands
    weights                  weights commands

optional arguments:
  -h, --help                 show this help message and exit
  -v, --version              show program's version number and exitDependenciestqdmnumpyscipypandasordered_set>=4.1.0ContributingIf you notice an error, please don't hesitate to open an issue.Development setup# updatesudoaptupdate# install Python 3.8, 3.9, 3.10 & 3.11 for ensuring that tests can be runsudoaptinstallpython3-pip\python3.8python3.8-devpython3.8-distutilspython3.8-venv\python3.9python3.9-devpython3.9-distutilspython3.9-venv\python3.10python3.10-devpython3.10-distutilspython3.10-venv\python3.11python3.11-devpython3.11-distutilspython3.11-venv# install pipenv for creation of virtual environmentspython3.8-mpipinstallpipenv--user# check out repogitclonehttps://github.com/stefantaubert/text-selection.gitcdtext-selection# create virtual environmentpython3.8-mpipenvinstall--devRunning the tests# first install the tool like in "Development setup"# then, navigate into the directory of the repo (if not already done)cdtext-selection# activate environmentpython3.8-mpipenvshell# run teststoxFinal lines of test result output:py38: commands succeeded
  py39: commands succeeded
  py310: commands succeeded
  py311: commands succeeded
  congratulations :)LicenseMIT LicenseAcknowledgmentsFunded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID 416228727 – CRC 1410CitationIf you want to cite this repo, you can use this BibTeX-entry generated by GitHub (seeAbout => Cite this repository).Changelogv0.0.3 (2023-05-30)ChangedImproved speed for filtering OOV/IV words by up to ~20k words/sAddedAddedsubsets select-randomlyAddedsubsets sort-by-shuffleAddedsubsets addoption--skip-existingBugfixFixed evaluation of "from subsets" to ensure that the subsets existFixedsubsets removedidn't workedv0.0.2 (2023-01-13)AddedAdded creation of weights from linesAdd--limitto select duplicatesAdd exit codeChangedSet--limitpositional where applicableDon't output expected warning fromnumpyon KLD selectionBugfixesv0.0.1 (2022-05-25)Initial release