Package: wikichangewatcher
WikiChangeWatcher 1.0.0Table of ContentsWikiChangeWatcher 1.0.0IntroductionInstallExamplesMonitoring “anonymous” page edits made from any IPv4 or IPv6 addressMonitoring “anonymous” page edits made from specific IP address rangesMonitoring page edits made by usernames that match a regular expressionMonitoring page edit events based on regular expression match on arbitary JSON fieldsCombining multiple filter classes with theFilterCollectionclassCombining/nesting multipleFilterCollectionclassesUsing bitwise AND/OR operators to createFilterCollectionclassesMonitoring “anonymous” edits made from IP address ranges owned by US government depts./agenciesCalculating a running average of page-edits-per-minute for all of wikipediawikiwatchCLI toolContributionsIntroductionWikipedia provides anSSE Streamof
all edits made to any page across Wikipedia, which allows you to watch all edits made to all wikipedia
pages in real time.WikiChangeWatcheris an SSE client that watches the SSE stream of wikipedia page edits,
with some filtering features that allow you to watch for page edit events with specific attributes
(e.g.“anonymous”edits with IP addresses in specific ranges, or edits made to a specific page, or edits made by a wikipedia
user whose username matches a specific regular expression).This package is inspired byTom Scott’s WikiParliament project.InstallInstall usingpip.pip install wikichangewatcherExamplesSome example scripts illustrating how to useWikiChangeWatcherare presented in
the following sections.Monitoring “anonymous” page edits made from any IPv4 or IPv6 addressThe following example code watches for edits made by any IPv4 or IPv6 address.# Example script showing how to use WikiChangeWatcher to watch for "anonymous" edits to any# wikipedia page from any IPv4 or IPv6 addressimporttimefromwikichangewatcherimportWikiChangeWatcher,IpV4Filter,IpV6Filter# Callback function to run whenever an event matching our IPv4 address pattern is seendefmatch_handler(json_data):"""
    json_data is a JSON-encoded event from the WikiMedia "recent changes" event stream,
    as described here: https://www.mediawiki.org/wiki/Manual:RCFeed
    """print("{user}edited{title_url}".format(**json_data))# Watch for anonymous edits from any IPv4  or IPv6 addresswc=WikiChangeWatcher((IpV4Filter()|IpV6Filter()).on_match(match_handler))wc.run()# Watch for page edits forever until KeyboardInterrupttry:whilewc.is_running():time.sleep(0.1)exceptKeyboardInterrupt:wc.stop()Monitoring “anonymous” page edits made from specific IP address rangesThe following example code watches for edits made by 3 specific IPv4 address ranges.# Example script showing how to use WikiChangeWatcher to watch for "anonymous" edits to any# wikipedia page from specific IP address rangesimporttimefromwikichangewatcherimportWikiChangeWatcher,IpV4Filter,IpV6Filter# Callback function to run whenever an event matching our IPv4 address pattern is seendefmatch_handler(json_data):"""
    json_data is a JSON-encoded event from the WikiMedia "recent changes" event stream,
    as described here: https://www.mediawiki.org/wiki/Manual:RCFeed
    """print("{user}edited{title_url}".format(**json_data))# Watch for anonymous edits from some specific IP address rangeswc=WikiChangeWatcher(IpV4Filter("192.60.38.225-230").on_match(match_handler),IpV6Filter("2601:205:4882:810:5D1D:BC41:61BB:0-ffff").on_match(match_handler))# Wildcard '*' character can be used in place of a IPv4 or IP46 address field, to ignore that field entirely.# IPV6 filter with some fields ignored: IpV6Filter("*:*:*:810:5D1D:BC41:*:0-ffff")# IPV6 filter with some fields ignored: IpV4Filter("192.*.*.225-230")wc.run()# Watch for page edits forever until KeyboardInterrupttry:whileTrue:time.sleep(0.1)exceptKeyboardInterrupt:wc.stop()Monitoring page edits made by usernames that match a regular expressionThe following example code watches for edits made by signed-in users with usernames
that contain one or more strings matching a regular expression.# Example script showing how to use WikiChangeWatcher to watch for NON-"anonymous" edits to any# wikipedia page, by usernames that contain a string matching a provided regular expressionimporttimefromwikichangewatcherimportWikiChangeWatcher,UsernameRegexSearchFilter# Callback function to run whenever an edit by a user with a username containing our regex is seendefmatch_handler(json_data):"""
    json_data is a JSON-encoded event from the WikiMedia "recent changes" event stream,
    as described here: https://www.mediawiki.org/wiki/Manual:RCFeed
    """print("{user}edited{title_url}".format(**json_data))# Watch for edits made by users with "bot" in their usernamewc=WikiChangeWatcher(UsernameRegexSearchFilter(r"[Bb]ot|BOT").on_match(match_handler))wc.run()# Watch for page edits forever until KeyboardInterrupttry:whileTrue:time.sleep(0.1)exceptKeyboardInterrupt:wc.stop()Monitoring page edit events based on regular expression match on arbitary JSON fieldsThe following example code watches for any page edit events where the specified JSON
field matches contains one or more matches of a regular expression (available
JSON fields and their descriptions can be foundhere).# Example script showing how to use WikiChangeWatcher to filter page edit events# by a regular expression match in an arbitrary named field from the JSON event# provided by the SSE stream of wikipedia page editsimporttimefromwikichangewatcherimportWikiChangeWatcher,FieldRegexSearchFilter# Callback function to run whenever an edit is made to a page that has a regex match in the page URLdefmatch_handler(json_data):"""
    json_data is a JSON-encoded event from the WikiMedia "recent changes" event stream,
    as described here: https://www.mediawiki.org/wiki/Manual:RCFeed
    """print("{user}edited{title_url}".format(**json_data))# Watch for edits made to any page that has the word "publish" in the page URL# ("title_url" field in the JSON object)wc=WikiChangeWatcher(FieldRegexSearchFilter("title_url",r"[Pp]ublish").on_match(match_handler))wc.run()# Watch for page edits forever until KeyboardInterrupttry:whileTrue:time.sleep(0.1)exceptKeyboardInterrupt:wc.stop()Combining multiple filter classes with theFilterCollectionclassThe following example watches for anonymous page edits to a specific page URL.# Example script showing how to use WikiChangeWatcher to watch for "anonymous" edits to# a specific wikipedia pageimporttimefromwikichangewatcherimportWikiChangeWatcher,FilterCollection,IpV4Filter,PageUrlFilter# Callback function to run whenever an event matching our filters is seendefmatch_handler(json_data):"""
    json_data is a JSON-encoded event from the WikiMedia "recent changes" event stream,
    as described here: https://www.mediawiki.org/wiki/Manual:RCFeed
    """print("{user}edited{title_url}".format(**json_data))# Default match type is is MatchType.ALLfilters=FilterCollection(# Filter for any edits to a specific wikipedia page URLPageUrlFilter("https://es.wikipedia.org/wiki/Reclus_(La_Rioja)"),# Filter for any IP address (any anonymous edit)IpV4Filter("*.*.*.*"),).on_match(match_handler)wc=WikiChangeWatcher(filters)wc.run()# Watch for page edits forever until KeyboardInterrupttry:whileTrue:time.sleep(0.1)exceptKeyboardInterrupt:wc.stop()Combining/nesting multipleFilterCollectionclassesThe following example watches for page edits to several specific page URLs made by
user with the word “bot” in their username.# Example script showing how to use WikiChangeWatcher to watch for edit to specific# wikipedia page URLs by users with the word "bot" in their nameimporttimefromwikichangewatcherimportWikiChangeWatcher,FilterCollection,UsernameRegexSearchFilter,PageUrlFilter,MatchType# Callback function to run whenever an event matching our filters is seendefmatch_handler(json_data):"""
    json_data is a JSON-encoded event from the WikiMedia "recent changes" event stream,
    as described here: https://www.mediawiki.org/wiki/Manual:RCFeed
    """print("{user}edited{title_url}".format(**json_data))# Make a filter collection that matches any one of several wikipedia pagespage_urls=FilterCollection(# Filters for any edits to multiple specific wikipedia page URLsPageUrlFilter("https://en.wikipedia.org/wiki/Python_(programming_language)"),PageUrlFilter("https://en.wikipedia.org/wiki/CPython"),PageUrlFilter("https://en.wikipedia.org/wiki/Server-sent_events"),).set_match_type(MatchType.ANY)# Make a filter collection that matches one of the page URLs, *and* a specific username regexmain_filter=FilterCollection(page_urls,UsernameRegexSearchFilter(r"[Bb][Oo][Tt]")).set_match_type(MatchType.ALL).on_match(match_handler)wc=WikiChangeWatcher(main_filter)wc.run()# Watch for page edits forever until KeyboardInterrupttry:whileTrue:time.sleep(0.1)exceptKeyboardInterrupt:wc.stop()Using bitwise AND/OR operators to createFilterCollectionclassesInstead of creating FilterCollection classes directly, you can instead use bitwise AND&and bitwise OR|to combine filter objects.For example, this code uses the bitwise OR operator to create a filter that matches any
IPv4 address,orany IPv6 address:fromwikichangewatcherimportIpV4Filter,IpV6Filter# Callback function to run whenever an event matching our filters is seendefmatch_handler(json_data):print("{user}edited{title_url}".format(**json_data))filter_collection=(IpV4Filter()|IpV6Filter()).on_match(match_handler)And this code creates an equivalent filter, but uses theFilterCollectionclass
directly instead:fromwikichangewatcherimportIpV4Filter,IpV6Filter,FilterCollection,MatchType# Callback function to run whenever an event matching our filters is seendefmatch_handler(json_data):print("{user}edited{title_url}".format(**json_data))filter_collection=FilterCollection(IpV4Filter(),IpV6Filter()).set_match_type(MatchType.ANY).on_match(match_handler)Finally, here is a slightly more complex example, which uses both bitwise AND / OR
operators together to create a filter that matches any IPv4 or IPv6 address,anda specific
page URL:fromwikichangewatcherimportIpV4Filter,IpV6Filter,PageUrlFilterPAGE_URL="https://en.wikipedia.org/wiki/Hayaguchi_Station"# Callback function to run whenever an event matching our filters is seendefmatch_handler(json_data):print("{user}edited{title_url}".format(**json_data))filter_collection=((IpV4Filter()|IpV6Filter())&PageUrlFilter(PAGE_URL)).on_match(match_handler)Monitoring “anonymous” edits made from IP address ranges owned by US government depts./agenciesThe following example watches for anonymous page edits toanywikipedia page,
from IP address ranges that were found to be publicly listed as owned by various
US government department and agencies (mostly California, some federal).If you want to look up some IP addresses owned by your local governments, or companies, it’s pretty easy,
I just went tohttps://ip-netblocks.whoisxmlapi.com/and searched for “california department of”
as the company name.# Example script showing how to use WikiChangeWatcher to watch for "anonymous" edits to any# wikipedia page from IP address ranges that are publicly listed as being owned by various US government departmentsimporttimefromwikichangewatcherimportWikiChangeWatcher,FilterCollection,IpV4Filter,IpV6Filter,MatchType# Callback function to run whenever an event matching one of our IPv4 address ranges is seendefmatch_handler(json_data):"""
    json_data is a JSON-encoded event from the WikiMedia "recent changes" event stream,
    as described here: https://www.mediawiki.org/wiki/Manual:RCFeed
    """print("{user}edited{title_url}".format(**json_data))filter_collection=FilterCollection(IpV4Filter("136.200.0-255.0-255"),# IP4 range assigned to CA dept. of water resourcesIpV4Filter("151.143.0-255.0-255"),# IP4 range assigned to CA dept. of technologyIpV4Filter("160.88.0-255.0-255"),# IP4 range assigned to CA dept. of insuranceIpV4Filter("192.56.110.0-255"),# IP4 range #1 assigned to CA dept. of correctionsIpV4Filter("153.48.0-255.0-255"),# IP4 range #2 assigned to CA dept. of correctionsIpV4Filter("149.136.0-255.0-255"),# IP4 range assigned to CA dept. of transportationIpV6Filter("2602:814:5000-5fff:0-ffff:0-ffff:0-ffff:0-ffff:0-ffff"),# IP6 range assigned CA dept. of transportationIpV4Filter("192.251.92.0-255"),# IP4 range assigned to CA dept. of general servicesIpV4Filter("159.145.0-255.0-255"),# IP4 range assigned to CA dept. of consumer affairsIpV4Filter("167.10.0-255.0-255"),# IP4 range assigned to CA dept. of justiceIpV4Filter("192.58.200-203.0-255"),# IP4 range assigned to Bureau of Justice Statistics in WAIpV6Filter("2607:f330:0-ffff:0-ffff:0-ffff:0-ffff:0-ffff:0-ffff")# IP6 range assigned to the US dept. of justice in WA).set_match_type(MatchType.ALL).on_match(match_handler)wc=WikiChangeWatcher(filter_collection)wc.run()# Watch for page edits forever until KeyboardInterrupttry:whileTrue:time.sleep(0.1)exceptKeyboardInterrupt:wc.stop()Calculating a running average of page-edits-per-minute for all of wikipediaThe following example watches for any edit to any wikipedia page, and updates a
running average of the rate of page edits per minute, which is printed to stdout
once every 5 seconds.# Example script showing how to use WikiChangeWatcher to watch for "anonymous" edits to any# wikipedia page from specific IP address rangesimporttimeimportstatisticsimportqueuefromwikichangewatcherimportWikiChangeWatcher# Max. number of samples in the averaging windowMAX_WINDOW_LEN=6# Interval between new samples for the averaging window, in secondsINTERVAL_SECS=5classEditRateCounter():"""
    Tracks total number of page edits per minute across all of wikipedia,
    using a simple averaging window
    """def__init__(self,interval_secs=INTERVAL_SECS):self._edit_count=0self._start_time=Noneself._interval_secs=interval_secsself._queue=queue.Queue()self._window=[]# Callback function to run whenever an edit event is seendefedit_handler(self,json_data):"""
        json_data is a JSON-encoded event from the WikiMedia "recent changes" event stream,
        as described here: https://www.mediawiki.org/wiki/Manual:RCFeed
        """self._edit_count+=1# Add an edit rate sample to the averaging window, and return the new averagedef_add_to_window(self,edits_per_min):self._window.append(edits_per_min)iflen(self._window)>MAX_WINDOW_LEN:self._window.pop(0)returnstatistics.mean(self._window)defrun(self):ifself._start_timeisNone:self._start_time=time.time()if(time.time()-self._start_time)>=self._interval_secs:# interval is up, calculate new rate and put it on the queueedits_per_min=float(self._edit_count)*(60.0/self._interval_secs)self._queue.put((self._add_to_window(edits_per_min),self._edit_count))self._edit_count=0self._start_time=time.time()defget_rate(self):ret=Nonetry:ret=self._queue.get(block=False)exceptqueue.Empty:passreturnret# Create rate counter class to monitor page edit rate over timeratecounter=EditRateCounter()# Create a watcher with no filters-- we want to see every single editwc=WikiChangeWatcher().on_edit(ratecounter.edit_handler)wc.run()# Watch for page edits forever until KeyboardInterrupttry:whileTrue:ratecounter.run()new_rate=ratecounter.get_rate()ifnew_rate:rate,since_last=new_rateprint(f"{rate:.2f}avg. page edits per min. ({since_last}in the last{INTERVAL_SECS}secs)")exceptKeyboardInterrupt:wc.stop()wikiwatchCLI toolA CLI program calledwikiwatchis provided, which uses thewikichangewatcherpackage to provide some monitoring capabilities at the command line:usage: wikiwatch [-h] [-a ADDRESS] [-u USERNAME_REGEX] [-f FIELD_NAME VALUE_RGX]
                 [-s FORMAT_STRING] [--version]

Real-time monitoring of global Wikipedia page edits, with flexible filtering
features.

options:
  -h, --help            show this help message and exit
  -a ADDRESS, --address ADDRESS
                        Adds an IPv4 or Ipv6 address range to look for. Any
                        anonymous edits made by IPv4 addresses in this range
                        will be displayed. Each dot-separated field (for IPv4
                        addresses) or colon-separated field (for IPv6 addresses)
                        may be optionally replaced with with an asterisk (which
                        acts as a wildcard, matching any value), or a range of
                        values. For example, the address range "*.22.33.0-55"
                        would match all IPv4 addresses in the range 0.22.33.0
                        through 255.22.33.50. This option can be used multiple
                        times to add multiple IP address filters.
  -u USERNAME_REGEX, --username-regex USERNAME_REGEX
                        Adds a username regex to look for. Any edits made by
                        logged-in users with a username that matches this
                        regular expression will be displayed. This option can be
                        used multiple times to add multiple username filters.
  -f FIELD_NAME VALUE_RGX, --field FIELD_NAME VALUE_RGX
                        Adds a regex to look for in a specific named field in
                        the JSON event provided by the wikimedia recent changes
                        stream (described here
                        https://www.mediawiki.org/wiki/Manual:RCFeed). Any edit
                        events which have a value matching the VALUE_RGX regular
                        expression stored in the FIELD_NAME field will be
                        displayed. This option can be used multiple times to add
                        multiple named field filters.
  -s FORMAT_STRING, --format-string FORMAT_STRING
                        Define a custom format string to control how filtered
                        results are displayed. Format tokens may be used to
                        display data from any named field in the JSON event
                        described at
                        https://www.mediawiki.org/wiki/Manual:RCFeed. Format
                        tokens must be in the form "{field_name}", where
                        "field_name" is the name of any field from the JSON
                        event. This option can only be used once (Default:
                        "{user} edited {title_url}").
  --version             Show version and exit.

NOTE: if run without arguments, then all anonymous edits (any IPv4 or IPv6
address) will be shown.

EXAMPLES:

Show only edits made by one of two specific IP addresses:

    wikiwatch -a 89.44.33.22 -a 2001:0db8:85a3:0000:0000:8a2e:0370:7334

Show only edits made by IPv4 addresses in the range 88.44.0-33.0-22:

    wikiwatch -a 88.44.0-33.0-22

Show only edits made by IPv4 addresses in the range 232.22.0-255.0-255:

    wikiwatch -a 232.22.*.*

Show only edits made by usernames that contain the word "Bot" or "bot":

    wikiwatch -f user "[Bb]ot"ContributionsContributions are welcome, please open a pull request athttps://github.com/eriknyquist/wikichangewatcher/pulls.
You will need to install packages required for development by doingpip install-rdev_requirements.txt.Please ensure that all existing tests pass, new test(s) are added if required, and the code coverage
check passes.Run tests withpython setup.py test.Run tests and and generate code coverage report withpython code_coverage.py(this script will report an error if coverage is below 90%)If you have any questions about / need help with contributions or tests, please
contact Erik ateknyquist@gmail.com.