Package: pytokr
pytokrVery simple, somewhat stoned tokenizer for teaching purposes.Current pip-installable version is 0.0.2 but the current status
of the repository is version 0.1.0.Behaviorally inspired by the early versions of theeasyinput module;
shares with it some similar aims, but not the aim of
conceptual consistency with C/C++.A separate, different evolution ofeasyinputisyogi.InstallThe usual incantation should work: pip install pytokr
(maybe with either "sudo" or "--user" or within a
virtual environment).If that does not work, download or clone the repo, then
put the pytokr folder where Python can see it from
wherever you want to use it.Simplest usage since version 0.1.0Finds items (simple tokens white-space separated) in a
string-based iterable such as stdin (default). Ends of
line are counted as white space but are otherwise ignored. Usage:from pytokr import pytokrthen call pytokr to obtain the tokenizer function; give it
whatever name you see fit, say,item:item = pytokr()Then, successive calls toitem()will provide you with
successive tokens fromstdin. In case no items remain,
an EndOfDataError exception will be raised. Note that,
as white-space is ignored, in case only white-space remains
then the programisat end of data.If a different source of items is desired, saysource(e.g. afilejustopen'ed or a list of strings),
simply pass it on:item = pytokr(source)In either case, a second output can be requested, namely, an
iterator over the items, say you want to name ititems:item, items = pytokr(iter = True)(such call would accept as well asourceas first parameter).
Then you can runfor itm in items():or make up als = list(items())and, with some care, avoid the dependence on the EndOfDataError
exception.Both combine naturally: the individual item function can be called
inside a for loop on the iterator, provided there is still
at least one item not yet read. That call will advance the
items; so, the next item at the loop will be the current one after
the local advances. Briefly: both advancethe sameiterator.All items provided are of typestrand will not contain
white space; casting intointorfloator whatever, if
convenient, falls upon the caller.ExampleBased onJutge problem P29448Correct Dates (and removing spoilers):from pytokr import pytokr
item, items = pytokr(f, iter = True)
for d in items():
    m, y = item(), item()
    if correct_date(int(d), int(m), int(y)):
        print("Correct Date")
    else:
        print("Incorrect Date")Deprecated usage of versions 0.0.*These versions were employed in a different manner. Version
0.1.0 can still be employed in the same way for some
backwards compatibility, but will print a deprecation
message tostderr. This old usage was:from pytokr import item, items(or only one of them as convenient). Thenitem()will provide
the next item instdinandfor w in items()will iterate on
whatever remains there. Callingitem()at end of file will
raise an exception EndOfDataError.Old, deprecated usage on other string-based iterablesAgain, this still works on 0.1.0 but will print a deprecation
message onstderr:from pytokr import make_tokrThen, ifgis an iterable of strings such as an open
file or a list of strings, the callitems, item = make_tokr(g)will provide adapted versions ofitemanditemsthat
will read them in fromginstead of fromstdin.