Package: ctparse
ctparse - Parse natural language time expressions in pythonFree software: MIT licenseDocumentation:https://ctparse.readthedocs.io.BackgroundThe packagectparseis a pure python package to parse time
expressions from natural language (i.e. strings). In many ways it builds
on similar concepts as Facebook’sducklingpackage
(https://github.com/facebook/duckling). However, for the time being it
only targets times and only German and English text.In principlectparsecan be used todetecttime expressions in a
text, however its main use case is the semantic interpretation of such
expressions. Detecting time expressions in the first place can - to our
experience - be done more efficiently (and precisely) using e.g. CRFs or
other models targeted at this specific task.ctparseis designed with the use case in mind where interpretation
of time expressions is done under the following assumptions:All expressions are relative to some pre-defined reference timesUnless explicitly specified in the time expression, valid resolutions
are in the future relative to the reference time (i.e.12.5.will
be the next 12th of May, but12.5.2012should correctly resolve
to the 12th of May 2012).If in doubt, resolutions in the near future are more likely than
resolutions in the far future (not implemented yet, but any
resolution more than i.e. 3 month in the future is extremely
unlikely).The specific comtravo use-case is resolving time expressions in booking
requests which almost always refer to some point in time within the next
4-8 weeks.ctparsecurrently is language agnostic and supports German and
English expressions. This might get an extension in the future. The main
reason is that in real world communication more often than not people
write in one language (their business language) but use constructs to
express times that are based on their mother tongue and/or what they
believe to be the way to express dates in the target language. This
leads to text in German with English time expressions and vice-versa.
Using a language detection upfront on the complete original text is for
obvious no solution - rather it would make the problem worse.Examplefromctparseimportctparsefromdatetimeimportdatetime# Set reference timets=datetime(2018,3,12,14,30)ctparse('May 5th 2:30 in the afternoon',ts=ts)This should return aTimeobject represented asTime[0-29]{2018-05-0514:30 (X/X)}, indicating that characters0-29were used in the resolution, that the resolved date time is the
5th of May 2018 at 14:30 and that this resolution is neither based on a
day of week (firstX) nor a part of day (secondX).Latent timeNormally,ctparsewill anchor time expressions to the reference time.
For example, when parsing the time expression8:00 pm, ctparse will
resolve the expression to 8 pm after the reference time as followsparse=ctparse("8:00 pm",ts=datetime(2020,1,1,7,0),latent_time=True)# default# parse.resolution -> Time(2020, 1, 1, 20, 00)This behavior can be customized using the optionlatent_time=False, which will
return a time resolution not anchored to a particular dateparse=ctparse("8:00 pm",ts=datetime(2020,1,1,7,0),latent_time=False)# parse.resolution -> Time(None, None, None, 20, 00)Implementationctparse- asduckling- is a mixture of a rule and regular
expression based system + some probabilistic modeling. In this sense it
resembles a PCFG.RulesAt the corectparseis a collection of production rules over
sequences of regular expressions and (intermediate) productions.Productions are either of typeTime,IntervalorDurationand can
have certain predicates (e.g. whether aTimeis a part of day like'afternoon').A typical rule than looks like this:@rule(predicate('isDate'),dimension(Interval))I.e. this rule is applicable when the intermediate production resulted
in something that has a date, followed by something that is in interval
(like e.g. in'May 5th9-10').The actual production is a python function with the following signature:@rule(predicate('isDate'),dimension(Interval))defruleDateInterval(ts,d,i):"""
  param ts: datetime - the current refenrence time
  d: Time - a time that contains at least a full date
  i: Interval - some Interval
  """ifnot(i.t_from.isTODandi.t_to.isTOD):returnNonereturnInterval(t_from=Time(year=d.year,month=d.month,day=d.day,hour=i.t_from.hour,minute=i.t_from.minute),t_to=Time(year=d.year,month=d.month,day=d.day,hour=i.t_to.hour,minute=i.t_to.minute))This production will return a new interval at the date ofpredicate('isDate')spanning the time coded indimension(Interval). If the latter does code for something else than
a time of day (TOD), no production is returned, e.g. the rule matched
but failed.Technical BackgroundSome observations on the problem:Each rule is a combination of regular expressions and productions.Consequently, each production must originate in a sequence of regular
expressions that must have matched (parts of) the text.Hence, only subsequence ofallregular expressions inallrules can lead to a successful production.To this end the algorithm proceeds as follows:Input a string and a reference timeFind all matches of all regular expressions from all rules in the
input strings. Each regular expression is assigned an identifier.Find all distinct sequences of these matches where two matches do not
overlap nor have a gap inbetweenTo each such subsequence apply all rules at all possible positions
until no further rules can be applied - in which case one solution is
producedObviously, not all sequences of matching expressions and not all
sequences of rules applied on top lead to meaningful results. Here thePCFG kicks in:Based on example data (corpus.py) a model is calibrated to
predict how likely a production is to lead to a/the correct result.
Instead of doing a breadth first search, the most promising
productions are applied first.Resolutions are produced until there are no more resolutions or a
timeout is hit.Based on the same model from all resolutions the highest scoring is
returned.CreditsThis package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject template.History0.3.4 (2022-07-28)Add fuzzy matching on longer literals[internal] De-tangle corpus tests into isolated test casesAllow spaces around separators in ruleDDMMYYYY and ruleYYYYMMDD0.3.3 (2022-07-18)Add rule for straight forward US formatted dates (ruleYYYYMMDD)Added ruleruleYearMonthAdded corpus cases for some open issues that now passChanged all internal imports to be absolute (i.e.from ctparse.xinstead offrom .x)Droppedtox(now using github actions)0.3.2 (2022-07-18)Drop support for python 3.6, update dev requirements0.3.1 (2021-07-07)Add support for python 3.9 on travis and in manifest; update build config0.3.0 (2021-02-01)Removed latent rules regarding times (latent rules regarding dates are still present)Added latent_time option to customize the new behavior, defauld behavior is backwards-compatible0.2.1 (2020-05-27)Update development dependenciesAdd flake8-bugbear and fixed issues0.2.0 (2020-04-23)Implemented new typeDuration, to handle lengths of timeAdapted the dataset to includeDurationImplemented basic rule to mergeDuration,TimeandIntervalin simple cases.Created a make target to train the modelmake train0.1.0 (2020-03-20)Major refactor of code underlying predictive modelBased on a contribution from @bharathi-srini: replace naive bayes from sklearn by own implementationThus remove dependencies on numpy, scipy, scikit-learnPredictions are much faster: 97/s in the old vs. 239/s in the new code basePerformance identicalDeprecate support for python 3.5, add 3.8Add more strict type checking rules (mypy.ini)Force black code formatting, make this a linter step, “black” all code0.0.47 (2020-02-28)Allow overlapping matches of regular expression when generating inital stack of “tokens”0.0.46 (2020-02-26)Implemented heuristics to detect (albeit imperfectly) military times0.0.44 (2019-11-05)Released time corpusImplemented training model using ctparse corpus0.0.43 (2019-11-01)Added slash as a general separatorAdded ruleTODTOD (to support expression like afternoon/evening)0.0.42 (2019-10-30)Removed nb moduleFix for two digit yearsFreshly retrained model binary file0.0.41 (2019-10-29)Fix run_corpus refactoring bugImplemented retraining utilities0.0.40 (2019-10-25)update develop dependenciesremove unused Protocol import from typing_extensions0.0.39 (2019-10-24)split ctparse file into several different modulesadded types to public interfaceintroduced the Scorer abstraction to implement richer scoring strategies0.0.38 (2018-11-05)Added python 3.7 to supported versions (fix on travis available)0.0.8 (2018-06-07)First release on PyPI.