Package: tagnews
tagnewstagnewsis a Python library that canAutomatically categorize the text from news articles with type-of-crime tags, e.g. homicide, arson, gun violence, etc.Automatically extract the locations discussed in the news article text, e.g. "55th and Woodlawn" and "1700 block of S. Halsted".Retrieve the latitude/longitude pairs for said locations using an instance of the pelias geocoder hosted by CJP.Get the community areas those lat/long pairs belong to using a shape file downloaded from the city data portal parsed by theshapelypython library.Sound interesting? There's example usage below!You can find the source code onGitHub.InstallationYou can installtagnewswith pip:pip install tagnewsNOTE:You will need to install someNLTKpackages as well:>>>importnltk>>>nltk.download('punkt')>>>nltk.download('wordnet')Beware,tagnewsrequires python >= 3.5.ExampleThe main classes aretagnews.CrimeTagsandtagnews.GeoCoder.>>>importtagnews>>>crimetags=tagnews.CrimeTags()>>>article_text=('The homicide occurred at the 1700 block of S. Halsted Ave.'...' It happened just after midnight. Another person was killed at the'...' intersection of 55th and Woodlawn, where a lone gunman')>>>crimetags.tagtext_proba(article_text)HOMI0.739159VIOL0.146943GUNV0.134798...>>>crimetags.tagtext(article_text,prob_thresh=0.5)['HOMI']>>>geoextractor=tagnews.GeoCoder()>>>prob_out=geoextractor.extract_geostring_probs(article_text)>>>list(zip(*prob_out))[...,('at',0.0044685714),('the',0.005466637),('1700',0.7173856),('block',0.81395197),('of',0.82227415),('S.',0.7940061),('Halsted',0.70529455),('Ave.',0.60538065),...]>>>geostrings=geoextractor.extract_geostrings(article_text,prob_thresh=0.5)>>>geostrings[['1700','block','of','S.','Halsted','Ave.'],['55th','and','Woodlawn,']]>>>coords,scores=geoextractor.lat_longs_from_geostring_lists(geostrings)>>>coordslatlong041.859021-87.646934141.794816-87.597422>>>scores# confidence in the lat/longs as returned by pelias, higher is betterarray([0.878,1.])>>>geoextractor.community_area_from_coords(coords)['LOWER WEST SIDE','HYDE PARK']LimitationsThis project uses Machine Learning to automate data cleaning/preparation tasks that would be cost and time prohibitive to perform using people. Like all Machine Learning projects,the results are not perfect, and in some cases may look just plain bad.We strived to build the best models possible, but perfect accuracy is rarely possible. If you have thoughts on how to do better, please considerreporting an issue, or better yetcontributing.How can I contribute?Great question! Please seeCONTRIBUTING.md.Problems?If you have problems, pleasereport an issue. Anything that is behaving unexpectedly is an issue, and should be reported. If you are getting bad or unexpected results, that is also an issue, and should be reported. We may not be able to do anything about it, but more data rarely degrades performance.BackgroundWe want to compare the amount of different types of crimes are reported in certain areas vs. the actual occurrence amount in those areas. In essence,are some crimes under-represented in certain areas but over-represented in others?This is the main question driving the analysis.This question came from theChicago Justice Project. They have been interested in answering this question for quite a while, and have been collecting the data necessary to have a data-backed answer. Their efforts includeScraping RSS feeds of articles written by Chicago area news outlets for several years, allowing them to collect almost half a million articles.Organizing an amazing group ofvolunteersthat have helped them tag these articles with crime categories like "Gun Violence" and "Drugs", but also organizations such as "Cook County State's Attorney's Office", "Illinois State Police", "Chicago Police Department", and other miscellaneous categories such as "LGBTQ", "Immigration".The web UI used to do this tagging was also recently updated to allow highlighting of geographic information, resulting in several hundred articles with labeled location sub-strings.Most of the code for those components can be foundhere.A group actively working on this project meets every Tuesday atChi Hack Night.See AlsoChicago Justice ProjectSource code of other CJP projects... including the database/web scraping side of thingsWhat is Chi Hack Night?