Package: twitter-fire-scraper
What is this?This is a Python Twitter "Fire event" scraper/listener.It is an application that will listen for or scrape data relating to house fires (Chicago specifically) in order to analyze how people use Twitter as a platform to report and talk about disasters.How will this use Twitter data?This application allows one to analyze, collect, and collate data about house fires and other disasters on Twitter.How do I install this?DependenciesPython 3 or greaterMongoDBStepspip install twitter-fire-scraperIf it is already installed and a newer version is available, you can update with:pip install twitter-fire-scraper --upgradeNotesThis README assumes all commands take place in the same folder as this README file.ExamplesExamples of how to use this package can be found inthis examples folderand also inour internal test suites.These should give you a good idea of how to use our scraper, and can be considered a 'living standard' of how our code works.Setting up your secretsThis secrets file is only used for the demos. When using this library, it is up to you to manage how you store and retrieve your API keys.More specifically, if theScraperobject is not initialized with aTwitterAuthenticationobject, it will search for a file called~/.twitterfirescraper/secrets.jsonfor API keys as a fallback.This is to make the demonstrations work and not recommended usage when using the library.A note: These are called 'secrets' for a reason. Don't ever stage or commitsecrets.json, please.Twitter secretsYou will need:A twitter developer account & API keyA consumer API key (goes into"consumer_key")A consumer API secret key (goes into"consumer_secret")An access token (goes into"access_token")An access secret (goes into"access_token_secret")A twitter handle you're authorized to make queries on behalf ofYou are to put these into a file calledsecrets.jsonin your home folder under.twitterfirescraper/(For example, mine isC:/Users/henryfbp/.twitterfirescraper/secrets.json.)An example file is provided for you to base your file off of, calledsecrets.example.json.MongoDB secretsThe demos in our code connect to the following mongodb address:mongodb://localhost:27017/Setting up a databaseFor the database, we have chosen to use MongoDB since twitter data is stored in JSON and MongoDB is very well-suited for storing JSON data.Followthis tutorialon how to install MongoDB.Developer dependenciesSame as above.Ruby, used for running scripts to build and test Python wheels.Setting up PipenvYou can install Pipenv by executingpip install pipenvYou can then install all packages (including dev packages liketwisted) in this folder's./Pipenvby executingpipenv install --devThen, you can run tests by executingpipenv run python /src/twitter-fire-scraper/tests/test/__main__.pyRunning a functional demoInside this folder, there are two files calledRun-Demo.batandRun-Demo.ps1. You can run either of those to start a demo intended for presentation purposes.Starting the Web APIThere is a web API that is included with thetwitter-fire-scraperpackage.It exposes functions of thetwitter-fire-scraperover HTTP.From sourceYou can run the web API from the live source code withpipenv run python twitter_fire_scraper/app.py.Using PyPIYou can run the Web API after installing it withpipby typingpython -m twitter_fire_scraper.appRunning testsYou can executepipenv run python fire-scraper/tests/<TESTNAME>.pyto run a test.To run all tests, executepipenv run python fire-scraper/tests/test/__init__.pyand all tests will run.Alternatively, if you have this package installed, runpython -m twitter_fire_scraper.tests.testto run the package's test module.What was this adapted from?A movie sentiment analysis project byRaul, the repositoryis hereand a live siteis here.Commit2fb844e8c081c1dc31cfb4760e3a80cefb6a0eeewas used.There's got to be a better way to run this than from the command line!There is! Use an IDE (like PyCharm, which I use) that preferably integrates with Python to show you import errors, syntax errors, etc. Go google "Python IDE" and pick one you like.Adding the location of Venv to your IDEIn order to run our tests through an IDE, we need to let our IDE know where venv was installed. I will explain this through Pycharm, but the method should be the same for any IDE.If runningpythonin windows powershell runs Python 3 (or you only have Python 3 installed), runpython -m pipenv --venvThis will yield the location of the python 3 Virtual Environment (It should be something likeC:\Users\Your Name\...\.virtualenvs\...). Copy this path and open Pycharm.Go intofiles -> settingsand expand theProject: fire-scraper-twitter. In the drop down, go intoProject Interpreter. Go to the top and click the gear and selectadd, as we will be adding a new interpreter.SelectExisting environmentand click the three dots to the right. Copy your path at the top, then OK everything.There! Done! Now we can run our tests from inside our IDE.Generating/uploading distribution archivesIf you want to distribute this source code as a Python Wheel, followthis guide.There are a series ofRubyscripts (cross-platform!) that handle building, cleaning, uploading.Make sure you have thetwinepackage installed for Python.Buildingruby build.rbCleaningruby clean.rbUploadingYou'll need to bump the version in./VERSIONwhen uploading a new version.To the test site (test.pypi.org)ruby upload.rb --testTo the real site (pypi.org)ruby upload.rb --deployTesting download and installThere are a couple ways for you to test how a user would experience installing this package.There are three Ruby scripts here, each doing what its name suggests.test-localwheel-install.rbwill install and test the latest WHL file generated bybuild.rb.test-testpypi-install.rbwill install and test the TEST PyPI'stwitter-fire-scraperpackage.test-realpypi-install.rbwill install and test the offical PyPI'stwitter-fire-scraperpackage.