Package: cranetoolbox
Crisis Racism and Narrative EvaluationProjectCRANE(Crisis Racism and Narrative Evaluation) aims tosupport researchers and anti-racist organisationsthat wish to usestate-of-the-art text analysis algorithmsto study how specificevents impact online hate speech and racist narratives. CRANE Toolbox is aPython package: once installed, the tools in CRANE are available as functions that users can use in their Python programs or directly through their terminal. CRANE targets users with basic programming butno machine learning skills.Skip to Quick-startTable of ContentsTable of ContentsIntroductionGenesisProject motivationCRANE Toolbox: The Python package for data analysisQuick-startPackage documentationHow to install and use CRANE ToolboxRequirementsInstall CRANE ToolboxCommand-line modeFunction-call modeModulesImport modulePreprocessing moduleAnalysis moduleVisualisation moduleContributingJoin the teamFrequently asked questionsOn the CRANE projectOn using CRANE ToolboxIntroductionBack to topGenesisCrisis Racism and Narrative Evaluation, orCRANEfor short, was born during theResiliency Challenge, a COVID19-driven hackathon. During nine weeks, an interdisciplinary team composed of students and professional designers, programmers and computer science researchers developed aprototype websiteforreal-time monitoring of racist rhetoric linked to the Covid19 pandemic. The project was proposed by Gianluca Stringhini, Boston University, following hiswork on early emergence of online sinophobic behaviour during the Covid19 (SARS-Cov-2) pandemic. Readers may find more details on the approach taken for the Resiliency Challenge on thecorresponding Devpost pageand theresiliency-challenge-legacy branchof this repository.Project motivationWhile doing literature review, we noticed that few papers on online hate speech monitoring had both a sociology research approach and recent natural language processing tools. The literature seems to be divided mainly into methodological papers aiming to develop new machine learning tools to detect and characterise racism, and traditional sociology research approaches, both qualitative and quantitative, that do not take full advantage of the available data and analysis methods.We are aware that this is due in part to the difficulty of gathering interdisciplinary teams from very different fields, and in part to the rather young and precarious nature of automated hate speech detection.Our experiences during the hackathon lead us to believe that these issues could be somewhat alleviated by providing social sciences researchers with user-friendly tools leveraging the more robust of the current hate speech detection and analysis methods.CRANE Toolbox: The Python package for data analysisCRANE Toolbox is designed tosupport researchers and anti-racist organisationsin studyinghow specific events impact online hate speech and racist narratives. State-of-the-art text analysis algorithms evolve quickly, and their spread is often limited to the dedicated field of computer science research. CRANE Toolbox wraps some of those methods, that fit typical social sciences questions, intouser-friendly tools that require no machine learning skills.:warning:In this first version, CRANE Toolbox will remainspecific to tweet analysis.CRANE Toolbox is aPython package. Onceinstalled, users cancall its functions in their Python scriptsor use the tools it providesdirectly through their terminal. Those tools are organised in several sub-packages called modules. Each module corresponds to a step in the general analysis pipeline.The first version of CRANE Toolbox provides tools and documentation for users to filter and format the data they have extracted from Twitter (importmodule), prepare it for analysis (preprocessingmodule), carry out the analysis(es) of their choice (analysis module), and get a basic graphic representation of the results (visualisationmodule).:warning: As a Python package, CRANE Toolboxdoes not provide the data to analyse. It also doesnot substitute for a research frameworkin which its tools should be used.  The documentation contains advice on parameter choices and results interpretation, but they remain the responsibility of the user.Quick-startBack to top:information_source: This section is for experienced programmers who want to dive head-first into the deep end. If you have no idea what we are talking about in there, please don't run away screaming!
Though the length of thePackage Documentation sectionmight make it a bit daunting, we wrote it with non-specialists in mind, so you should find all the instructions and advice you need to use the CRANE Toolbox.InstallpipinstallcranetoolboxTransform DataThe first step is to transform your data into our common data exchange format, a three column CSV file.
We use thecrane-importmodule to perform this step. It takes source files that contain a single JSON tweet object per line and transforms them into a lightweight CSV file. The available parameters are detailed in theImport Modulesection.crane-import--source-folder./my_source--output-folder./my_outputPreprocess DataIt is important to clean-up text data before data analysis. We provide a CLI tool to perform this step, with several preprocessing options. The available parameters are detailed in thePreprocessing Modulesection.crane-preprocess./my_output./my_preproc_outputRun analysisWe currently provide a single statistical analysis tool that performs basic frequency analysis on the data set using a provided set of keywords. The required input formats and available parameters are detailed in theAnalysis Modulesection.crane-analysis-quanti./my_preproc_outputkeywords.jsonquanti_results.csvPackage documentationBack to topThis section is divided into two parts: a first one describing how to install CRANE Toolbox and run its tools, either in command-line mode in a terminal or through function calls in Python scripts; and a second one giving details on each module, in particular data format and parameter choice.Advanced users can find additional information on the content of each module in thecode documentation.How to install and use CRANE ToolboxRequirementsIn order to successfully install CRANE Toolbox, you will need:A basic internet connectionA terminalPython 3pip for Python 3Somebasic knowledge in programming and command-lines is strongly recommended, but the brave and the studious may choose to proceed without.If the above list is clear for you, skip to theinstallation instructions. Else, read the following paragraphs to get your environment ready.:warning:If you choose not to follow the recommended installation process, you will probably need to install additional packages or software.What is a terminal and where do I find one?The terminal is an interface in which you can type and execute text based commands. You may find a more detailed explanationhereand pictures of what a terminal looks likehere.A terminal is already installed on your computer. If you are using a Unix-based operating system (eitherMacOSX or a Linux distribution), it is aptly namedTerminaland lives with your other applications. If you are using a Windows operating system, it is namedCommand Promptand can be found in your Start Menu. For Windows 8 and 10, it is located in theWindows Systemfolder.To run a command in the terminal, simply type it in and press Enter.:warning: Copy-pasting shortcuts (Ctrl^C and Ctrl^V) are disabled by default in terminals, because Ctrl^C is used to abort programs instead. Use right-click to copy and paste.How do I install Python 3?Python 3is a programming language. :warning: It is very similar to Python 2 but the two are not interchangeable.The installation process depends on your operating system. Basic instructions can be found onthe official Python website, but you will easily find more detailed guides on the internet.How do I install pip for Python 3?pipis a package manager: it helps you download Python packages and update them when necessary.Basic instructions for its installation can be found onits official website. It is widely used so step-by-step guides and other resources are easily found on the internet.:warning: Because of the co-existence of Python 2.7 and Python 3, pip might be installed for both languages on your computer. In that case, use thepip3command instead. You can check which version ofpipis used by runningpip --versionin your terminal (python -m pip --versionfor Windows users).Install CRANE ToolboxWe strongly recommend installing CRANE Toolbox usingpip. Runpip install cranetoolboxin a terminal.
The command might take some time becausepipwill also automatically install all the
Python packages required for CRANE Toolbox to run.
You can check the list of dependencies on the [PyPi page of CRANE Toolbox](link to Pypi page).CRANE Toolbox can also be installed manually from the source code available in this repository.Command-line modeThe command line mode is the easiest method of using this package, it enables the user to quickly run all parts of the package and produce usable statistics in a very short period of time.There are three different CLI commands available in the toolbox, they arecrane-import,crane-preprocessandcrane-analysis-quanti. In order, they handle importing/transforming raw data into a standard CSV format, preprocessing that data into a usable format and finally computing the daily frequencies of given keywords.For specifics on the parameters provided under each CLI tool please check the detailed documentation below.Function-call modeThis package has been written with reuse in mind. While the CLI tools are easier to use as standalone tools, CRANE Toolbox can be imported and used within another python project. All functions are available to the user through standard Python package usage. In addition, a single high level function corresponds to each CLI tool so users can easily call their desired pipeline directly from Python.ModulesImport moduleThe import module is accessible from thecrane-importcommand-line entry point.The goal of this module is totransform raw twitter JSON data into something more manageable and lightweightfor use by the rest of the pipeline.This is not a required stepand could be performed manually by the user or have already been done via some other tool since data is passed between modules as files.This module canprocess a large number of large (multiple gigabyte) fileswithout relying heavily on swap or memory usage. This is accomplished by reading files by chunks (the user can specify the number of lines in the options) as well as writing in chunks.
This ensures that only a certain amount of data is held in memory at any given moment.In addition, it removes extra tweet data that is not required for the analysis pipeline, reducing file size and increasing the performance of later steps in this package.Expected input formatThe module will attempt to read any file in the specified folder, regardless of extension, name, etc. It supports any text-based file format (.json, .csv, .txt). In addition, it can handle compressedtararchives and treat multiple files or folders in a given tar archive. :warning: Currently, it only supports singly compressed tar archives -- thus notar.gzor similar formats are supported.Regardless of the exact file type, datashould always be stored with oneJSON object tweetper line, using\nas the end-of-line character.Output FormatThe module will concatenate all files into a single CSV file with the following columns:idtextcreated_atunique id of tweetfull text tweettimestamp of tweetThe CSV file hasno headers, comma separators, and optional double quotes for text.The text column is dynamically selected depending on if the tweet is over 140 characters or not, for more information about extended tweets see theofficial documentation on extended tweetsIf the output file already exists, it will append to the existing file. In the case that it does not exist it will create a new one, using the default path ("./filtered_data.csv") if none is specified.CLI CommandsThe import package can be used as a command-line tool and supports a several different options.(Required)--source-folderThe source folder or file to scan for files/archives to process.(Optional)--output-folderThe folder to save the output. The specified folder must exist. Defaults to./.(Optional)--output-nameThe name, including extension, of the output file. Defaults tooutput.csv.(Optional)--text-nameThe name to text field, case this field has a different name.(Optional)--date-nameThe name to created_at field, case this field has a different name.(Optional)--id-nameThe name to id field, case this field has a different name.(Optional)--tweet-languageThe language of tweets saved to the file. Based on the language field in the
JSON object. Defaults toen.(Optional)--max-lines-in-memoryThe maximum number of lines that will be held in memory. This can be adjusted to
to optimize for performance or on machines that have limited memory. Defaults to50000.(Optional)--retweetsUse this flag toincluderetweets in the output set. Defaults tofalseA complete example for the command-line entry-point:crane-import--source-foldertweets/november--output-foldermydataset/data--max-lines-in-memory2000--output-namedataset.csvPreprocessing moduleThe preprocessing pipeline is accessible from thecrane-preprocesscommand-line entry point.The proposed preprocessing is adapted fromEffrosynidis et al. (2017), using techniques 9, 0, 1, 3, 5 and 7, and 4.Expected input formatThe dataset given as the first positional argument can be either a folder of CSV files or a single CSV file. Each CSV file should contain3 columns(ID of input, should be unique, int; Text of the tweet, string; timestamp: Timestamp string),no headers, comma separators, and optional double quotes for text.idtextcreated_atunique id of tweetfull text tweettimestamp of tweetPreprocessing stepsIn order and for a given string, it will:Transform uppercase characters to lowercase,Remove escaped unicode strings (e.g.\u002c),Remove non-ascii characters,Replace urls with "url", or remove them if-urlis specified,Replace mentions (e.g. "@BobTheSurfer") with "atUser", or remove them if-mentionis specified,Remove the hash symbol in front of hashtags (e.g. "#spreadTheLove", and segment the hashtags if-hashtagis specified,Replace contractions with their full spelling (e.g. "I'm" becomes "I am"),Replace repeated punctuation signs with a textual equivalent (e.g. "!!!" becomes "multiExclamation") and newlines with spaces, or remove all punctuation except underscore characters ifpunctis specified,Replace numbers with their English text version, or remove them if-numis specified.Hashtags are segmented,i.e.separated into words, using thewordsegmentpackage. :warning: Only supported for English language. :warning::warning::warning::warning:Word segmentation runs in polynomial time and increase the preprocessing time by several orders of magnitude.:warning: The algorithm to unpack contractions is currently rather basic. It will for example mistakenly interpret the possessive "'s" as a verb contraction.:warning: Ordinal numbers are not yet supported.Output FormatThe processed dataset is saved in the folder given as the second positional argument. If it does not exist, the folder will be created. For each input file a processed file is generated. Files names are generated by appending "_preprocessed" to the name of the corresponding input file. Each CSV file contains4 columns(ID of input, should be unique, int; Original text of the tweet, string;Preprocessed text of the tweet, string; timestamp: Timestamp string),no headers, comma separators, and optional double quotes for text.idoriginal_textclean_textcreated_atunique id of tweetfull text tweetpreprocessed text of tweettimestamp of tweetCLI CommandsThe pipeline has two mandatory positional arguments and five optional arguments:(Required) Position 1. Path to the folder containing the dataset formatted with theimportmodule, or a single dataset file.(Required) Position 2. Path to the folder to save the preprocessed dataset in. If it does not exist, the folder will be created.(Optional)-urlor--remove-urlUse this flag to remove URLs from the tweets instead of replacing them with 'url'.(Optional)-mentionor--remove-mentionsUse this flag to remove user mentions '@userHandle' from the tweets instead of replacing them with 'atUser'.(Optional)-hashtagor--segment-hashtagsUse this flag to segment hashtags instead of simply removing the preceding '#' character.(Optional)-punctor--remove-punctuationUse this flag to remove all punctuation expect hyphens, instead of replacing repeated symbols and newlines.(Optional)-numor--remove-numbersUse this flag to remove all numbers from the tweets instead of replacing them with their text version.A complete example for the command-line entry-point:crane-preprocessmydataset/datamydataset/preprocessedData-punctAnalysis moduleCurrently the analysis module only offers some simple quantitative analysis pipeline to compute the daily frequency of given keywords.Simple quantitative analysisThis analysis pipeline is accessible from thecrane-analysis-quanticommand-line entry point.It computes the daily frequency of given keywords in a dataset. It allows for variants of keywords. For example, occurrences of "boys" and "boyz" can be counted together.Expected input formatThe dataset given as the first positional argument can be either a folder of CSV files or a single CSV file. Each CSV file should contain4 columns(ID of input, should be unique, int; Original text of the tweet, string; Preprocessed text of the tweet, string; timestamp: Timestamp string),no headers, comma separators, and optional double quotes for text. The preprocessed text is assumed to be lower-case.idoriginal_textclean_textcreated_atunique id of tweetfull text tweetpreprocessed text of tweettimestamp of tweetThe keywords are defined in a JSON dictionary where the keys are the main variant for each keyword and the values are lists of variants. All keywords should be lower-case strings. For example:{"color":["colour","color"],"chinese":["chinese","chineze","chines"]}Output formatThe output is a CSV file with adaydate column (format "%Y-%m-%d"), atotal_countcolumn with the daily total number of tweets in the dataset, a[keyword]_countcolumn for each keyword (its main variant is used to name the column) with the daily number of tweets containing at least one variant of the keyword and a[keyword]_freqcolumn for each keyword (its main variant is used to name the column) with the daily frequency of tweets containing at least one variant of the keyword.For example, for the keywords set given above:daytotal_countcolor_countchinese_countcolor_freqchinese_freqdatedaily number of tweetsdaily number of tweets with "color" or a variantdaily number of tweets with "chinese" or a variantdaily frequency of tweets with "color" or a variantdaily frequency of tweets with "chinese" or a variantCLI CommandsThe pipeline has three mandatory positional arguments and one optional argument:(Required) Position 1. Path to the folder containing the dataset preprocessed with thepreprocessmodule, or a single dataset file.(Required) Position 2. Path to the JSON file containing the keywords and their variants. See below for the expected format.(Required) Position 3. Path for the result file.(Optional)-dor--date-formatString defining the format of dates in the dataset. The default is %a %b %d %H:%M:%S %z %Y".A complete example for the command-line entry-point:crane-analysis-quantimydataset/preprocessedDatakeywords.jsonquanti_results.csv-d"%d %b %a %h:%M:%S %z %Y"Visualisation moduleNot implemented yetContributingBack to topWe welcome all contributions! If you have questions, a feature request or some feedback, please use theissuesfeature of GitHub.TheIssues pageworks rather like your good old forum page, with many additional features specific to programming and git versioning. GitHub provides adetailed guideto them, but here are the basics:Search the existing issuesfor similar questions/feature requests/feedbackIf you find an issue similar enough, you can add to its discussion with your own detailsIf you don't find an issue similar enough, create a new oneGive as much relevant information as possible. For example, if you want to report a bug or get help with an error, provide both your code/command-line and the error message. If possible, link to the data.Add appropriate labels/tags to the issue. For example, "documentation" if you're missing info in the doc to use the toolbox.Always keep in mind that this is a volunteer project, and contributors do their best to help.Be nice and patient.Adding to the code baseYou are welcome to submit Pull Requests for open issues. In particular, issues tagged "help-wanted" are usually things the core team is struggling with.
Here are the guidelines we ask you to follow when contributing to the code base.Comment on the issue to notify everyone of your intention to do so to make sure no one else is working on the same problem.Tackle only one issue per PR, unless you've discussed it with the core team before and they agree a grouped PR makes more sense.Reference the issue you are tackling in your PR.Tag your PR with "need-review" when making your initial submission or after completing the changes requested by your reviewers.Check your PR for reviews and be open to suggestions. If a reviewer is requesting changes, they will change the label of your PR from "need-review" to "in-progress".Join the teamIf you wish to involve yourself further (reviewing PRs, planning for new features, researching machine learning methods, doing user research, ...), you can join the core team by emailing bolduc2 (at) hotmail (dot) fr to get onboarded. We welcome developers, of course, but also designers, researchers from all academic fields, technical writers...We have chosen to onboard people privately instead of sharing all our resources in the repo for two reasons. First, we wanted to make it easier for users not used to open source projects to find what they need. Second, some of our resources contain private information from user research.Frequently asked questionsBack to topOn the CRANE projectWhat is this website linked to the repo?https://crane-toolbox.github.io/#/is a prototype website build during the Resiliency Challenge hackathon, in an attempt to quantify and characterise the impact of the COVID19 pandemic on online sinophobic hate speech. We left it online as a very basic example of the type of data analysis that could be carried out with CRANE Toolbox.Are you going to continue working on real-time analysis of COVID19-related online hate speech?We do not plan to at the moment. Due to the composition of the team, we have decided to focus our efforts on CRANE Toolbox, where we believe we can make a difference.Who is doing this?LaChapeliere💻⚠️📖🤔🚧📆Gianluca Stringhini🤔Marko Shiva Pavlovic💻Claudio Catterina💻⚠️SvetlanaMd🖋📖🎨📓🤔Ian💻⚠️📖🤔🚇📆👀Paul Han💻Kelly💻🤔🐛Gabriel Ribeiro💻Isaac📖Tomáš Lokša💻📖Judith van Stegeren💻Emoji keyThis project follows theall-contributorsspecification. Contributions of any kind welcome!The list of hackathon contributors for the first phase of the project can be found onCRANE Devpost page.Can I help?Yes! Check out theContributingsection.On using CRANE ToolboxWhat is Python? What is a Python package? What is a Python module?Python is a very common programming language for data analysis tasks. Packages are installable add-ons to the basic Python language. Roughly speaking, a module is a file with Python code.That said, if you are entirely unfamiliar with Python, you might want to experiment with it some before using CRANE Toolbox.I have a question/a feature request/some feedback, how do I contact you?Please use theIssuespageof this repo. Check out theContributingsection for more details.Can I copy your code for my project?CRANE Toolbox is distributed underGNU Affero General Public License v3.0.You can use it however you want provided you respect the license requirements: include a license and copyright notice, state the changes you made, disclose your source (this repo), and distribute your code under the same license. Please note that we are not liable for whatever use you make of this code, and we provide no warranty.How do I cite CRANE Toolbox in my paper?Please link to this repo.Where can I find Twitter data?Sharing of tweets datasets is restricted byTwitter Developer Terms. You will not be able to find publicly available datasets with tweets content, but rather datasets of tweet ids. Those can behydrated, using for exampleHydrator.From there it really depends on your research topic and questions.
Some are very time-sensitive, some are less. If yours are not very time-sensitive, try your luck with a regular search engine.
Some require anaivedataset, representative of the full information flow of Twitter. In this case, your best bet is to collaborate with a research team doing frequent analyses on Twitter content. They probably keep a stream running to collect 1% of the daily Twitter data.
Others require a dataset that has already been filtered for specific keywords or users. If that's your case, try your luck with a regular search engine.
If you don't need historical data, you could also start your own Twitter stream to collect exactly the data you desire.My data is not in your specific input format, what do I do?We are working to include several standard input formats to ourimportmodule. If your data format is not supported, pleaseget in touchand we will do our best to either include support for it in CRANE Toolbox or guide you to transform it into one of our supported formats.Method X is very useful and should be included, why is it not there?Maybe we don't know about it, maybe we didn't have time to implement it yet, maybe we choose not to include it for a given reason. Pleaseget in touchto tell us about it. (Unless it's a proprietary method with a use fee.)