fixed typo in analyze old: ana cursor, not clean cursor.
back to work after christmas
back to work after christmas
fixed cleaning problem. articles now being cleaned. used False, when None was there.
fixed cleaning problem. articles now being cleaned. used False, when None was there.
one, perhaps very large, article caused problems for clean. used janitor to delete that article. clean now works again, with 2 articles at a time.
simplified memcache method for clean.
working on simplified memcache for janitor
working on simplified memcache for analyze, then splitting analyze into several files
working on adding a backend. remember to actually appcfg update the backend.
lots of confusion with backends. still no msft in db.
msft now being added to db! dynamic backend.
backend db insertion test works with resident backends, not dynamic like you though. that's good. you want resident.
scraping now handled by backends, without errors, but with five yellow spikes every hour.
changed the order of the catch-all in app.yaml, and juggled versions in order to make the logs reflect the correct name of the backend.
deleted old task queue, which was full of tasks that were handled by the frontend. these were the calls to backend_scraping and /scrape, and they caused the yellow spikes, since they all failed.
running duplicate companies on backend. no errors. still sometimes deadlineexceeded, but sometimes more milliseconds without deadlineexceeded.
possibly fixed scrape and dupes with memcache. now testing.
apparently, i have working scrape and dupes. clean is next.
backend now serving scrape and clean. dupes only by title check in scrape.