Package: tsl-compiler
TSL â€“ Text Scraping LanguagePython package for processing of a scraping language in pseudo-codeThe TSL Python package allows you to write and execute pseudo-code style language to process text files with Regular expressions and simple logic. This gives an easy entry to data mining to non-programmers.You can either run it as a CLI usingpython TSL.py myScript.tslor use theTSLEngine classlike this:fromTSLEngineimportTSLEngineTSL=TSLEngine('myScript.tsl')ifTSL.task:TSL.run()Example:... This will read all lines fromstats/milestones.csv, take all lines, splits them into columns, selects the second column and saves the corresponding row into a file labeled by said column (e.g.stats/31-03-2019.txt).IndexHow does it work?SetupAvailable TSL CommandsTemplatingHow does it work?TSL runs through the script line by line and executes corresponding Python code in the background. File handling, complex data types, and templating are built-in for rapid prototyping. Every line starts with a command followed by a space and space-separated arguments.
Most commands support optional clauses likeas ...(storage variable) orin ...(file handle) to supply further information.A command's inputs and outputs can bestringsorcollections of strings. In ladder case, TSL iterates over a collection's strings and applies the command to each of them. The commandsas,remember,split, andfor everyloops change the context to the provided variable. This means you can omitasclauses in the following commands, always automatically referring to the context. To reference variables rather than strings use square brackets.log somethingwill log the string "something", whilelog [something]will log the content of the variable calledsomething.SetupUsepip install tslto install the package.Available TSL CommandsFile & system operationsbash<command>as<variable>Runs a bash command and saves the returned output to a variable.Example:bashgitbranchasbranchesempty[<filepath>]Opens up a file and deletes all its content.Example:inwordbag.txtemptyin<path/to/textfile.txt>Opens up a file and reads all its lines. You can log the lines usinglog lineAll future file operations are refering to this one until your next "in" statement.
You'll usually see this followed by atakeorfind allcommandExample:instats/01092019.txtin<path/to/folder>Creates the nested directory structure if it doesn't exist. Otherwise, the path will be used as context for future operations.Example:in"/Sublime Text/Packages"countfilesasfileCountlog[fileCount]save[as <filepath>]Saves the latest collection in the given filename.Example:saveasrunner/cleaned_userinputs.txtwrite[<variable>]Writes given variable (or the results of the lastfind all) into the last file opened withinExample:write[userIds]add<string | variable>[to<filepath>]Appends content to a file different from the currently open oneExample:add[libraries]tolibs.txtSelectionsselect nth [of[input]]Selects a specific item of a collection, given its index.Example:inbigrams.txtselect4thselect words [of[input]][as<output>]Selects all words found in the last opened file.Example:inutterances.txtselectwordsselect [from<string | RegEx | int>] [to<string | RegEx | int>]Selects the range from the indicated string/RegEX/number until the indicated string or regular expression or number. Note that we start counting with 1 to keep it naturalExample:selectfrom"accessibilityApp"to"[v:"selectfrom\sto\sselectfrom1to"[v:samsung.tvSearchAndPlay.Genres:drama]"selecttwoof[bigrams]select from<string | RegEx | integer>Selects the range from the indicated string / regular expression / number until the end of the line**Example:selectfrom"dateTime"selectfrom\d\d\dselectfrom122select to<string | RegEx | integer>Selects the range from the beginning of the line to the indicated string / regular expression / number.Example:selectto"dateTime"selectto\Wselectto5thselectto370Debugging & calculationsbe<property>Sets one of the following properties of TSL to true:verbose|activecalculateoperationas<variable>Calculates mathematical operationsExample:calculate(5*4)/2asratiolog<variable | string>Prints to the console. Use strings with template tags (e.g. "here is: [varName]") for variablescount<variable>as<countVariable>Stores the count of lines in a selection.Example:count[entries-per-day]asfrequencylog[frequency]count<files | folders>in<path/to/dir>as<countVariable>Stores the count of files or folders in a directory.Example:countfilesin"C:\Windows"assystemFileslog"Exactly [systemFiles] system files found."Manipulationchange<varName>to<formula>Iterates over a collection and changes all entries according to the template tag. Use brackets to tag variables, like so:[varName]Example:change[salute]to"Hi, [salute] #[i]"will e.g. change "my name is Dan" to "Hi, my name is Dan #1"combine<setName>with<setName>as<varName>Merges two sets and stores it in a new variable.Example:combine[vowels]with[consonants]aslettersfind all<string | RegEx>[in<varName>] [as<varName>]Finds all occurrences of a string or regular expression in the lines of the currently open file or a stored collection. The results of this search are automatically stored in a variablefoundExample:incorpus_de.txttakelinesasutterancesfindall[aeiou]+in[utterances]log[found]remove linesRemoves the last selected lines (e.g. the ones found using afind all)replace<string | RegEx>by<string>[in<variable>]Replaces given string or regular expression by another string, optionally in a particular collection.Example:replace\W+by"_"sort [<varName>]Sorts either the supplied or last referenced collection alphanumerically (in ascending order).split<string|RegEx>by<delimiter>as<variable>Splits a string into a collection using delimiter.Example:splitapples;bananas;orangesby;asfruitslog[fruits]unique linesRemoves all duplicate lines from the last referenced collection.Memoryremember<string | variable>as<variableName>Stores a string or variable in a new variable.take<lines | results | files | folders>[as<name>]Changes the selected collection to whole lines (take lines as ...), results of afind alldirective, or to the files found in a folder specified with a precedingin <folderPath>directive.Example:insource.txtfindall<[^>]+>takelinesashtmlLineslog[htmlLines]inlibraries/detakefilesasgermanLibslog[germanLibs]Flowfor every<variable>---Loops through a collection, populating the variableiwith the current index. From within the loop, the item of the collection can be accessed using the variable name in singular (books -> book, babies -> baby).If a collection is empty, the for-loop is skipped. This becomes useful to create conditional flows.Always terminate a loop with three consecutive hyphens in a separte line.Example:incorpus.txtfindall[^\b]+\b[^\b]+asbigramsforevery[bigram]log"#[i]: [bigram]"---runpath/to/script.tslRuns another TSL fileThe external TSL file will receive the same scope as inlined code.TemplatingTemplates are enclosed in square brackets and can appear in quoted strings, file paths, and even within regular expressions:{remember"\CommNetwork"asdomaininuser.txtfindall\b[domain][^:]:asuserforevery[user]selectfrom0to-1in"/users/[user]/credentials.txt"change[user]to"[user]:pleaseresetme"add[user]---}If the variables can not be found, the template tags remain untouched, including square brackets. This allows us to easily mix them in with regular expressions.