Package: royston
RoystonAn end-to-end machine learning library for detect trending stories and content in real time. An open source Python framework that currently works in memory on a single node and can comfortably perform with around 500k-1m articles. Parallelisation is very much on the near term road map (next 6 months).Trends are identified by detecting phrases that start occurring much more frequently than those that don't typically occur. Various natural language processing and data science techniques are used to ensure similar words are modelled together (i.e. "cycle", "cycling" and "cyclist" all reduce down to a common word form, such as "cycle").Documents can be grouped by a subject, so it is possible to detect "localised" trends. It is often the case that a trending story has a number of related phrases (for example, "doping scandal" and "Tour de France"), so this is handled using hierachical clustering and doc2vec to handle this.Based onramekin, but going to take it further to do real time detection and maintaining models rather than creating them each time.Installation and basic usageWe are going to create a royston to contain a set of news articles, and then find the trends.First we will install the package viapipby typing the following into the command line:pip3 install roystonThe following script creates some simple documents and adds them to aroyston(also shipped in theexamplesdirectory):from royston import Royston
from datetime import datetime as dt

roy = Royston()

# ingest a few documents
roy.ingest({ 'id': '123', 'body': 'Random text string', 'date': dt.now() })
roy.ingest({ 'id': '456', 'body': 'Antoher random string', 'date': dt.now() })

# find the trends - with this example, it won't find anything, as it's
# only got two stories!
trends = roy.trending()
print(trends)Configuration OptionsConstructor:This package is heavily configurable to allow us to tune how we look for emerging trends. The default options have been set for the most common use case that looks at new trends that have emerged over the last 24 hours.Currently, the main way of tuning these parameters is controlled by passing the Royston constructor anoptionsdict with the following attributes:AttributeTypeDefaultDescriptionmin_trend_freqint4A threshold for the minimum number of times a phrase has to occur in a single day before it can even be considered a trend for a given subject.history_daysint90The context of the number of days to consider for the history. This means we look at how often a phrase has occured over this period, and get an idea of typical use.trend_daysint1The period of time in which we want to look for trends. With the default of 1, we are looking at documents from the last day to see if new trends have emerged during that time compared with the typical use period defined byhistory_daysmax_nint6The maximum size of the n-gram window (i.e. the window size of each phrase)history_frequency_tolerancefloat1.6Factor the history count by this amount to handle words that just didn't get mentioned in the history period. This usefulness of this is in review, and it is likely to be removed in future (or at least set to 1 by default).trends_top_nint8The maximum number of trends to returnDisclaimer: the following options are currently supported but expected to change significantly in future releases:AttributeTypeDefaultDescriptionstartdatetimenow - trend_daysThe start of the "trend" period (i.e. a day ago)enddatetimenowThe end of the "trend" periodhistory_startdatetimestartStart of the trend period (i.e.history_daysbeforeend)history_enddatetimeend- history_daysStart of the trend period (i.e.history_daysbeforeend)Currently they are calculate in the constructor only, which is stupid, as we want this to run in realtime and adapt each time thetrendmethod is called.Running testspoetry run testRun coverage reports:poetry run coverageDistributeThis now uses poetry for package management, which can be done with the following command:poetry build && poetry publishContribute?This is still in the early stages of being ported over from JavaScript, and any help would be appreciated. The issues contain a lot of features that are needed. Please get in touch viaLinkedInand I can talk you thought anything.Main concerns are:100% test coverage.Retain the document formatCode formatted using black/flake8