Package: ebdata
ebdataCode to help write scripts that import/crawl/parse data from the web
intoebpub,
as well as extract (US) street addresses from (English) text.This package is part of OpenBlock. Originally developed for EveryBlock.com.For more information, see thedocumentationor theproject website.Problems can be reported to theissue tracker.Discussion is on theebcode google groupor the #openblock channel on freenode.InstallationDonotjust try to easy_install or pip install ebdata.  It has a lot
of specific dependencies which can’t/shouldn’t be captured by
setup.py.Instead, see the full documentation athttp://openblockproject.org/docs/install/index.htmlwhich includes links to pip requirements files
and instructions on preparing your system.OpenBlockOpenBlock is a web application and RESTful service that allows users
to browse and search their local area for “hyper-local news” - to see
what’s going on recently in the immediate geographic area.For installation instructions and other documentation, seehttp://openblockproject.org/docs/(or the .rst files in the docs/ directory).For help, you can try the ebcode group:http://groups.google.com/group/ebcodeor look for us in the #openblock IRC channel on irc.freenode.net.About the ProjectOpenBlock began life as the open-source code released by
Everyblock.com in June 2009.  Originally created by Adrian Holovaty
and the Everyblock team, it is now developed as an open-source (GPL)
project by OpenPlans, athttp://openblockproject.org.Funding for the initial creation of Everyblock and the ongoing
development of OpenBlock has been provided by the Knight Foundation
(http://www.knightfoundation.org/).OpenBlock 1.2.0 (Released 5/23/2012)Upgrade NotesAs usual, install all dependencies, eg if you are upgrading a source checkout:pip install -r ebpub/requirements.txt
pip install -e ebpub
pip install -r ebdata/requirements.txt
pip install -e ebdata
pip install -r obadmin/requirements.txt
pip install -e obadmin
pip install -r obdemo/requirements.txt
pip install -e obdemoAs usual, sync and migrate the database:django-admin.py syncdb
django-admin.py migrateIn this release, there are some changes to the Block model and the
import script, which combine to give you better search results and
geocoding if you reload your blocks data.  To do so:Run migrations, as described aboveShut down your scrapersBack up your databaseRe-import blockswith the –reset option(that’s important!), via either
the admin UI or the command line … seehttp://openblockproject.org/docs/install/geodata.html#streets-blocksBe sure to also regenerate the Streets, BlockIntersections, and
Intersections as per the docs.Browse your site… everything should work except that streets
with previously cryptic names like “99” or “87” will now have a
more recognizable full name, eg. “State Route 99” or “Interstate
87”.Start your scrapers again.The website should remain operational during this process, though
some pages may briefly give errors during the process, and URLs for
highways will change to include the prefix.Backward IncompatibilitiesRemoved foreign keys EmailAlert.block and SavedPlace.block; instead
they now have geometries which can be used to find a nearest block.
This makes it safer to re-load your blocks data if you need
to. Running migrations will take care of existing alerts and saved
places, but you can’t safely downgrade if you change your mind; make
database backups as always!  (Ticket #257)Removed mystery fields Schema.grab_bag, Schema.grab_bag_headline,
Schema.intro. (Ticket #232)Removed safe_id_sort_reversed template tag; use the for loop’s
reverse option instead, eg.
{% for item in itemlist|safe_id_sort reversed %}Moved the friendlydate template filter from ebpub.db.templatetags.eb
into ebpub.db.templatetags.dateutils, where it seems to belong.Moved the recaptcha template tag from ebpub.neighbornews.templatetags into
ebpub.db.templatetags.recaptcha_tags.
Templates using it will now need to do {% load recaptcha_tags %}.Moved one obscure template tag, {% get_locations_for_item %},
from ebpub.widgets.templatetags into ebpub.db.templatetags.recaptcha_tags.
Templates using it will now need to do {% load eb %}.Renamed a bunch of template tag functions to match the name of the
tag, eg. “do_filter_url()” is now “filter_url()”.  This makes the
API docs easier to read; it doesn’t affect templates, only code that
imports those functions directly - and there problably is none of
that.Added a “View selected items on map” link and checkboxes on Schema
filter pages, to allow viewing explicitly selected items on the “big map”.Changed URLs used by the schema_filter view, so some bookmarks may
break. (Ticket # 266)Removed the EB_MEDIA_ROOT and EB_MEDIA_URL settings; now use
django’s normal MEDIA_ROOT and MEDIA_URL instead.Removed theImproperCityexception, which only served to prevent
using blocks in places such as unincorporated parts of counties,
where there is nothing that could be called a ‘city’.Removed unused NewsItem.block field, part of ticket #93.Removed ebdata/retrieval/scrapers/new_newsitem_list_detail.py,
which wasn’t used anywhere.Removed ebpub/streets/blockimport/tiger/import_blocks2010.py.
Census 2010 files are now supported by the main import_blocks.py
script.Removed item.intersecting from Widgets context; it was never
documented properly. Instead use the newget_locations_for_itemtemplate tag.Removed the old map javascript since we’re now using
openblockrichmap.js everywhere.New Features in 1.2You can now safely re-import blocks data.  In fact, this is highly
recommended, you’ll get better geocoding results due to fixes for
#264 and #278.  Use the “–reset” option to “import_blocks_tiger”.Added ebpub.utils.geodjango:interpolate() function which
finds the point on a linestring that’s the given distance (or
fraction) from the first point.  (Adds dependency on Shapely,
since otherwise this would require a database call.)Added ‘prefix’ to the Streets models, used for things like “State Hwy” in the
US census TIGER data; without it, we were getting geocoding failures,
confusing/ambigous street names in the UI, etc. Closes #264.Location import in admin UI now runs in background.Admin UI now provides links to view various things (NewsItems,
Locations, Blocks, Streets, LocationTypes) on the live site.NewsItemnow allows saving with an emptydescription.;
some things might really only have a title available.Support multiple types of Yahoo maps, due to olwidget upgrade.ebpub.geocoder.base.full_geocode() now has a convert_to_block
argument, factored out from ebdata.retrieval.  If True, this
tries to disambiguate bad blocks on good streets by rounding down
to the nearest block, eg. converting ‘299 Wabash St’ to ‘200 block
of Wabash St.’  May help geocoding when eg. census data doesn’t
quite match reality.Schemas now have anedit_windowfield, representing how long (in
hours) users are allowed to edit their content after it’s created.  Used
by theneighbornewsforms.Rest API: Allow searching by multiple types (schemas).Added an admin UI for importing NewsItems from spreadsheets
(currently only handles CSV and old-style Excel sheets; not .xslx)
(Ticket #126)Added a generic spreadsheet scraper inebdata/scrapers/general/spreadsheet/retrieval.py,
(currently only handles CSV and old-style Excel sheets; not .xslx)
(Ticket #274)ebdata.scrapers.general.georssaddress-extraction fallback now
looks in all tags that look like text.Search form now searches Places too.Neighbornews schemas now have chartablecategories.Allow overriding the template for schema_filter view on a per-schema
basis, by creating a template named db/schema_filter/<schema>.htmlNieghbornews schemas now have specific templates for the
schema_filter view.Addedfeaturedflag onebpub.db.Lookupmodel, allowing admins
to designate some Lookup values as “special”, for use in
eg. navigation. (#268)Added aget_featured_lookups_by_schematemplate tag, puts into
context a list of the “special” Lookup values for that schema. (#268)AddedLookup.objects.get_featured_lookup_for_item(newsitem, attribute_key)method to find out which “featured” Lookups a newsitem has for a
given attribute.Added alookup_values_for_attributetemplate tag, dumps all
values of a given db.attribute field as a JSON list.Made schema_filter the default view of Schemas, ticket #272Addedebpub.moderationapp that allows users to flag NewsItems
as spam or inappropriate, and an admin UI for it;
seemoderationfor more.AddedSchema.allow_flaggingboolean to toggle moderation
flagging per schema. Allowed by default on theebpub.neighbornewsschemas.User-uploaded images now supported for NewsItems, and enabled for
theebpub.neighbornewsuser-contributed content schemas.Added newuserlinks.htmltemplate so you can override the links
at top right of the page.Added aget_locations_for_itemtemplate tag, see../main/widgetsfor more.Now works with Postgresql 9.1, ticket #262Nicer map controls thanks to Frank Hebbert, ticket #225Added advanced hook for filtering schemas based on arbitrary request
data; implement this by assigningsettings.SCHEMA_MANAGER_HOOK = 'some_module:some_function', wheresome_module.some_functiontakes
arguments (request, manager) and returns amodels.Managerinstance
whose query sets will return the allowed Schemas.Add Vary headers to REST API responses, for more correct HTTP
cache-ability.Auto-complete categories on the “neighbornews” add/edit forms.Optional ReCaptcha on the user-contributed (“Neighbornews”) add/edit
forms.User-contributed content (“neighbornews”) now has edit and delete forms.Sensible defaults on most DateFields and DateTimeFields, can still
be overridden.Logout now redirects you to whatever page you were viewing.Add a “properties” JSON field to the Profile model, for more
flexible per-user metadata.User admin UI now shows Profiles and API keys inline.“Sticky widgets” aka “pinned” NewsItems in widgets: You can use the
admin UI to make certain NewsItems stay visible in the widget
permanently or until an expiration date that you set.settings.NEIGHBORNEWS_USE_CAPTCHA can now be a string path to a
function.New NewsItem.objects.by_request() method for filtering based on
eg. user privileges.New get_schema_manager(request) method for filtering based on
current request, with an extensibility hook too.At least put the darn geocoder cache results in the admin so you can
delete them manually if desired. Refs #163Admin UI option to save a copy of a schema as a new schema.Bugs fixedAdmin users no longer have a limit on file upload size.Handle importing locations in other projections than 4326.
Thanks Karen Tracey. Refs #214.We now use the primary street name for blocks loaded from US census TIGER
data, instead of whichever one we see last, which may be an alternate name.
Issue #278Location import (both command-line and admin UI) no longer blows up
when re-importing the same Locations.RSS feed URLs fixed to use settings.EB_DOMAIN rather than the sites
framework for getting the root URL.  For consistency with the rest
of OpenBlock.Street names properly normalized on input (ticket #166)parsing.normalize() and text.slugify() no longer blow up if fed
non-string input.Schemas with allow_charting=False were shown on the schema_filter
view, but not on its map. Fixed schema_filter_geojson so now they
show up on map too.Fix filtering by location and date on big map page.Fix #281, wrong schemas shown on big map page.Map icon URLs for db.Location and streets.PlaceType can now be
relative to STATIC_URLFix #282, missing items on place detail pagesFix KeyError when an Attribute references a non-existent Lookup.Fix error on FilterChain.add(key, lookup) when key isn’t a SchemaField.Should be possible to run OpenBlock at a URL prefix now; removed all
hardcoded URLs. Ticket #90.Fix missing AJAX timeouts on “save place” button, thanks Tim Shedor.
Ticket #269Fix error in NewsItem.objects.by_attribute() with many-to-many
lookups: looking for [3,47] was finding any number starting with 3
or ending with 47.Makemanage.shscript executable.Fix rare error when we have a Block instance but its block range
doesn’t match the block range regex. Known example: 1600-7-1600-9
Hanover Blvd. in Columbia, MO.Allow choosing multiple values when filtering via Lookups.
Ticket # 267.Use query params instead of weird URIs for schema_filter view,
ticket # 266.Remove bogus breadcrumbs from schema_filter page; ticket #270Filtering NewsItems by Block no longer causes 500 error.block_import_tiger can now be safely re-run on the same file,
it won’t create duplicate blocks anymore.Fixed double-logging of scrapers to the console./streets/ list doesn’t blow up if you haven’t set
DEFAULT_LOCTYPE_SLUG.Workaround for getting profile when request.user is a LazyUser
instance.De-hardcoded more URLs.When using a too-old python version, our setup.py scripts now give a
more informative error, instead of SyntaxError due to awithstatement.Custom login view now works when going to admin site, and is
compatible with (uses same cookies as) django.contrib.auth. Ticket
#174Logout form was broken by bad template name. Fixed.Fix 500 error when user doesn’t exist.Don’t barf constructing richmaps url if there are no matching
newsitemsGroup blocks by street on “choose a block” page, ticket # 263Store suffixes on streets with names like ‘Wilson Park’; fixes some
geocoding failures.Fix 500 error on newsitem.geojson, ticket #38DocumentationToo many to mention, but here are some:Python API documentation for all(?) of ebpub, ebdata, obadmin, obdemo packages.
Ticket #159.Better docs on how to import spreadsheets via scraper or admin.Clarify behavior of email alerts.Better docs on what the “core” NewsItem fields are.Documentebpub.db.binscripts. Ticket #96.Documentation about comments and flagging of NewsItems. Ticket #252Better docs about template overrides, seecustom-look-feel.Documentebpub.streets.Places, seeplaces.  Ticket #253Basic docs forebpub.neighbornews, seeuser_content.
Ticket #211Document how to get the 2010 census files instead of 2009.Added docs on all the settings in settings_default.py.Better documentation about Schemas, SchemaFields, Attributes, and how they relate.Fixes to example crontab, thanks Tim ShedorOtherFactored out the georss scraper’s point-parsing code into aget_point()function in ebdata.retrieval.utils.Generic rss scraper is now the basis forobdemo.scrapers.add_newswhich did the same thing.Generic rss scraper is now a ListDetailScraper and
RssListDetailScraper subclass. Ticket #242Upgrade jquery-ui to 1.8.17.Upgrade jquery to 1.7.1.Moved some NewsItemListDetailScraper functionality up into
BaseScraper, so it’s more widely usable.Deprecate log_exception(), the logging module actually does that
alreadyMove full_geocode() to ebpub.geocoder.base;  it was in an obscure placeBy default, one API key per user.  3 was kind of silly.