Package: tweet-parser
Authors:Fiona Pigott,Jeff
Kolb,Josh
Montague,Aaron
GonzalesGoal:Allow reliable parsing of Tweets delivered by the Gnip platform, in both
activity-streams and original formats.Status:This package can be installed by cloning the repo and usingpip install-e., or by usingpip install tweet_parser.As of version 1.0.5, the package works with Python 2 and 3, and the
API should be relatively stable. Recommended to use the more recent release.
Current release is 1.13.2Currently, this parser does not explicitly support Public API Twitter
data.Usage:This package is intended to be used as a Python module inside your other
Tweet-related code. An example Python program (after pip installing the
package) would be:fromtweet_parser.tweetimportTweetfromtweet_parser.tweet_parser_errorsimportNotATweetErrorimportfileinputimportjsonforlineinfileinput.FileInput("gnip_tweet_data.json"):try:tweet_dict=json.loads(line)tweet=Tweet(tweet_dict)except(json.JSONDecodeError,NotATweetError):passprint(tweet.created_at_string,tweet.all_text)I’ve also added simple command-line utility:pythontools/parse_tweets.py-f"gnip_tweet_data.json"-c"created_at_string,all_text"Testing:A Pythontest_tweet_parser.pypackage exists intest/.The most important thing that it tests is the equivalence of outputs
when comparing both activity-streams input and original-format input.
Any new getter will be tested by runningtest$ python test_tweet_parser.py, as the test checks every method
attached to the Tweet object, for every test tweet stored intest/tweet_payload_examples. For any cases where it is expected that
the outputs are different (e.g., outputs that depend on poll options),
conditional statements should be added to this test.An option also exists for run-time checking of Tweet payload formats.
This compares the set of all Tweet field keys to a superset of all
possible keys, as well as a minimum set of all required keys, to make
sure that each newly loaded Tweet fits those parameters. This shouldn’t
be run every time you load Tweets (for one, it’s slow), but is
implemented to use as a periodic check against Tweet format changes.
This option is enabled with--do_format_validationon the command
line, and by setting the keyword argumentdo_format_validationtoTruewhen initializing aTweetobject.ContributingSubmit bug reports or feature requests through GitHub Issues, with
self-contained minimum working examples where appropriate.To contribute code, fork this repo, create your own local feature
branch, make your changes, test them, and submit a pull request to the
master branch. The contribution guidelines specified in thepandasdocumentationare a great reference.When you submit a change, change the version number. For bug fixes and
non-breaking changes that do not affect the top-level Tweet object API
(fixing a bug or changing the internals of a getter while package naming/structure
remains the same), increment the last number (X.Y.Z -> X.Y.Z+1) insetup.py. For changes that do affect the top-level Tweet object API (e.g., adding a
new getter), increment the middle number (X.Y.Z -> X.Y+1.0).Guidelines for new gettersAgetteris a method in the Tweet class and the accompanying code in
thegetter_methodsmodule. A getter for some property should:be named<property>, a method inTweetdecorated with@lazy_propertyhave a corresponding method namedget_<property>(tweet)in thegetter_methodsmodule that
implements the logic, nested uner the appropriate submodule (a text
property probably lives under thegetter_methods.tweet_textsubmodule)provide the exact same output for original format and
activity-streams format Tweet input, except in the case where certain
information is unavailable (seeget_poll_options).In general, prefer that theget_<property>work on a simple Tweet
dictionary as well as a Tweet object (this makes unit testing easier).
This means that you might useis_original_format(tweet)rather thantweet.is_original_formatto check format inside of a getter.Adding unit tests for your getter in the docstrings in the “Example”
section is helpful. See existing getters for examples.In general, make detailed docstrings with examples inget_<property>, and more concise dosctrings inTweet, with a
reference for where to find theget_<property>getter that
implements the logic.StyleAdhere to the PEP8 style. Using a Python linter (like flake8) is
reccomended.For documentation style, useGoogle-style
docstrings.
Refer to thePython docstest
documentationfor
doctest guidelines.TestingCreate an isolated virtual environment for testing (there are currently
no external dependencies for this library).Test your new feature by reinstalling the library in your virtual
environment and running the test script as shown below. Fix any issues
until all tests pass.(env)[tweet_parser]$pipinstall-e.(env)[tweet_parser]$cdtest/;pythontest_tweet_parser.py;cd-Furthermore, if contributing a new accessor or getter method for payload
elements, verify the code works as you intended by running theparse_tweets.pyscript with your new field, as shown below. Check
that both input types produce the intended output.Note that FieldDeprecationWarnings will appear while testing for certain getters, this is expected behavior.(env)[tweet_parser]$pipinstall-e.(env)[tweet_parser]$pythontools/parse_tweets.py-ftest/tweet_payload_examples/activity_streams_examples.json-c<yournewfield>And lastly, if you’ve added new docstrings and doctests, from thedocsdirectory, runmake html(to check docstring formatting)
andmake doctestto run the doctests.