dobtco/openrfps — pull request #5 by jasonlally (2014-02-24 06:06:20+00:00)
Title: Add San Francisco scraper

This is pulling down the Office of Contract Admin's list of bids. There doesn't seem to be any record of bid awards in this system (but you can get that data out of the vendor file maintained by the Controller, I believe).

So this is just existing bids. Some issues/questions:

1. **Unique IDs:** There is no consistent unique ID across bids, the Bid Number varies based on originating department, which is an artifact of SF's pretty decentralized and autonomous departments. Also there was at least one duplicate bid listed currently and many without any Bid Number whatsoever, so QA seems to be in question. To avoid ID conflicts, I've appended the database resource ID from the query parameters of the url after a dash.
2. **No created_at, updated_at:** There are no updated or created dates that I can see, but there is a flag to indicate Changes accompanied by a text description of the changes. There is very little structured here, but I could plug in an updated flag with just true or false as the options, if that would be helpful?
3. **New fields:** I added a couple of fields that aren't in the schema currently. Let me know if you want them removed? They are: estimate, duration, contact_fax
4. **Time zones:** I've hard coded a timezone, but PST will give way to PDT, so probably not a best practice? Should I just drop a timezone altogether and instead let that be inferred programmatically?

Thanks for doing this, by the way!