Package: efnlp
efnlpThis is thejankiest possibletake at computing n-gram or conditional empirical frequency ((C)EF) distributions for successor tokens given a fixed-width prefix. Yes that's just a Markov model. In a way this ispurposefullynaive, and interested in how far such a naive technique goes.There'spythonandc++code (inefnlpandcpprespectively) for analyzing text and creating sampling datastructures. There's alsogocode (ingo) for a gRPC server to read sampling datastructures and serve batch or streaming generated text.Motivation (even though we just playin)The basic idea is to explore computing these values and using them in generative sampling tasks. We "hypothesize" that the asymptotic (in data and model class universality) result of computational(L)LMsis a high-fidelity representation of the (C)EF in the training data. This is a bit tautologous, for a consistent statistical model estimated on sufficient data. Seepaper/*.pdffor more; however I have had repeated personal experiences with really cool complicated models basically reducing to the information in counts and averages, much like (C)EFs.Of course, a (C)EF "model" should have serious generalizability challenges when confronted with valid sequences not appearing in the training data. Particularly if we wish to generate text (via token sequences), we may generate sequences purely by chance that have no (or limited) precedent and thus have limited predictive capacity. (A Markov model can always reduce to token occurrence frequencies, though randomly choosing tokens is far from language modeling.) This is a primary goal of statistical models: to find (or use) structure to "fill in the gaps" of a particular corpus of training data. However, as a training corpus grows to encompass more and more of the possibilities of discourse, the likelihood we have not "seen" a valid sequence should also decrease, and a model should also be getting progressively more aligned with the (C)EFs.Also we would expect the required storage for a purely (C)EF "model" to grow quite large. This is, in principal, another primary goal of statistical models: to "compress" statistical information without diminishing validation/generalization performance. Yet, the SOTA statistical models are themselves quite extraordinarily large; samples are not reasonable computable on a single machine, involving many billions of parameters and linear algebra involved enough to require specialized hardware.Obviously if simple (C)EF models were of comparable quality to modern LLMs they would be making all the noise instead. This isn't about a better approach, just a personal investigation.ExampleYou can run thepythoncode from the CLI as$python-mefnlp-cdata/tinywillspeare.txt-m-s-b10-g100000-osample-results.txt[2023-01-28T21:37:56.223686|0.000020s]Readingtextcorpus[2023-01-28T21:37:56.225040|0.001374s]Forming(character)language[2023-01-28T21:37:56.273636|0.049969s]Encodingcorpusinthelanguageconstructed[2023-01-28T21:37:56.345648|0.122002s]Corpusis1,115,393tokenslong[2023-01-28T21:37:56.345711|0.122036s]Parsingprefix/successortokens[2023-01-28T21:38:24.269634|28.045963s]Parsedprefixesandsuccessorsincorpusin27923.90ms[2023-01-28T21:38:27.680794|31.457128s]Found858,920prefixes(77.0%ofpossible)[2023-01-28T21:38:34.836321|38.612656s]Found937,254patterns(84.0%ofpossible)[2023-01-28T21:38:37.554198|41.330531s]Memory(roughly)required:31.9MB(about4,177,347dbl,8,354,695fl)[2023-01-28T21:38:37.554237|41.330562s]Samplinganddecoding100000tokens[2023-01-28T21:38:38.493836|42.270165s]Generationfrequency:9.4us/tok[2023-01-28T21:38:38.493869|42.270194s]Writingsampledresultstosample-results.txt(for which you can see generated text inresults). The compiledc++is similar,cpp$./efnlp++-cdata/tinywillspeare.txt-m-s-b10-g100000-osample-results.txtor thegogo$gorun*.go-parse\-input../data/tinywillspeare.txt\-language../cpp/language.proto.bin\-block10\-generate10000\-print=false(note the use of alanguagedatastructure inprotoformat).Our "model" here (inpython) is, more or less, 8Mdoubles worth of "parameters" and "trains" (estimates) in a single process on an (old) macbook in under a minute (for 10-token sequence statistics). Sampling is basically constant time, relying on hashmaps; the example above takes about 0.1ms per character sampled (with zero code optimizations). The (C)EF "model" are a significant inflation of the data size: 1.1MB of data turns into 62.4MB of statistics. But honestly the results aren't that bad. It's junk of course, but on the surface comparable to generative text from a 10M parameter transformer style model applied to the same dataset that trained for 15 minutes on a GPU (tweet here,code here).The full CLI arg list can be reviewed from$python-mefnlp-hMore comprehensive results are detailed in the following tables (seepaper/*.pdffor more discussion):B# prefixesunique pref.# patternunique patt.avg succ./pref.memory1650.0%1,4030.1%21.63kB21,4030.1%11,5561.0%8.236kB311,5561.0%50,7124.5%4.4221kB450,7124.5%141,02112.6%2.8876kB5141,02112.6%283,31325.4%2.02.5MB7447,35240.1%609,65954.7%1.410.1MB10858,92077.0%937,25484.0%1.131.9MB12991,39188.9%1,027,85792.2%1.050.4MB151,069,42395.9%1,081,06096.9%1.080.6MB201,103,35898.9%1,106,34599.2%1.0133MBWhat follows are comparative parsing/generation speeds (for sampling 1M tokens):Bpy parsepy t/τgo parsego t/τc++ parsec++ t/τ11.0s1.4μs49ms0.1μs131ms0.1μs22.0s1.7μs109ms0.1μs248ms0.2μs33.3s2.1μs222ms0.2μs419ms0.3μs44.3s2.6μs361ms0.3μs612ms0.4μs56.4s3.2μs585ms0.5μs1.1s0.5μs712.0s4.9μs1.2s0.7μs2.0s0.7μs1028.0s7.0μs2.6s0.9μs1.9s0.8μs1237.3s8.3μs4.1s1.1μs2.5s1.0μs1554.3s9.7μs5.2s1.2μs3.2s1.0μs20129.0s12.7μs8.4s1.6μs4.4s1.3μsNote (until we get better formatting) that the repeated "parse" (parse time) and "t/τ" (time per token) columns are forpython,go, andc++respectively. With each we can generate text at O(μs)/token. However compiled codes (goandc++) are faster (roughly by an order of magnitude) and seem to scale a bit better with longer sequences. With compiled code we can parse out the (C)EFs in shakespeare in seconds; whilepythonis still "reasonable" it takes a couple minutes to parse 20-token sequences.UsagepythonSeeefnlp/__main__.pyfor an example of running the no-doubt-dumb implementations inefnlp/__init__.py.c++Seecpp/*, with stuff that should be ready forcmaketo run through a build. Embarassingly, pretty much everything is just inmain.cpp.goSeego/*, with stuff that should be ready to build. There's a very briefREADME.md