diaspora/diaspora — pull request #3482 by Raven24 (2012-08-03 23:01:39+00:00)
Title: fist working version of a 'post fetcher' for remote accounts

This is my initial "basic functionality first, chocolate sprinkles later" version of a resque job which fetches the public posts from a remote account and adds them to the local database.
In the current configuration it is being run after a webfinger requests in order to populate a newly fetched profile with the person's existing public posts, that would otherwise never have gotten to the local pod. But it would also be possible to trigger the job in other places where it would make sense...

This patch includes a database migration to add the "fetch_status" column to the people table, in order to record wether a person's posts have already been fetched, or if resque is currently working on it.

While fetching, the code makes heavy use of the federation logger and of course I also included a huge pile of rspec tests, which should provide full coverage.

In my small little "lab" tests everything seems to be working OK, but I would like everyone to review the flaming hell out of this. It should be pretty self-contained and worst-case this would introduce a few (hundred) more failed resque jobs (depending on how many webfinger requests a pod does...).

here my federation logger output:

```
integration2 - 1344033529.696 [fyi] (pid:27869) webfingering user1@localhost:3001, it is not known or needs updating
integration1 - 1344033529.720 [fyi] (pid:25832) webfinger profile request for :28
integration2 - 1344033530.480 [fyi] (pid:27869) successfully webfingered user1@localhost:3001
integration2 - 1344033536.110 [fyi] (pid:27873) fetching public posts for user1@localhost:3001
integration2 - 1344033536.295 [meh] (pid:27873)
[{"id"=>105, "guid"=>"3c6a9385c0b0310a", "text"=>"and now for the thrid and last post", "public"=>true, "created_at
"=>"2012-08-03T22:37:46Z", "interacted_at"=>"2012-08-03T22:37:46Z", "provider_display_name"=>nil, "post_type"=>"Sta
tusMessage", "image_u
integration2 - 1344033536.346 [fyi] (pid:27873) saving fetched post (3c6a9385c0b0310a) to database
integration2 - 1344033536.346 [meh] (pid:27873)
{"id"=>105, "guid"=>"3c6a9385c0b0310a", "text"=>"and now for the thrid and last post", "public"=>true, "created_at"
=>"2012-08-03T22:37:46Z", "interacted_at"=>"2012-08-03T22:37:46Z", "provider_display_name"=>nil, "post_type"=>"Stat
usMessage", "image_ur
integration2 - 1344033536.523 [fyi] (pid:27873) saving fetched post (632a5d3ce152b361) to database
integration2 - 1344033536.523 [meh] (pid:27873)
{"id"=>104, "guid"=>"632a5d3ce152b361", "text"=>"this is the second one in a series of very interesting posts", "pu
blic"=>true, "created_at"=>"2012-08-03T22:36:31Z", "interacted_at"=>"2012-08-03T22:36:31Z", "provider_display_name"
=>nil, "post_type"=>"
integration2 - 1344033536.557 [fyi] (pid:27873) saving fetched post (66c90f6feaea5450) to database
integration2 - 1344033536.557 [meh] (pid:27873)
{"id"=>103, "guid"=>"66c90f6feaea5450", "text"=>"this is my first public post", "public"=>true, "created_at"=>"2012
-08-03T22:34:50Z", "interacted_at"=>"2012-08-03T22:34:50Z", "provider_display_name"=>nil, "post_type"=>"StatusMessa
ge", "image_url"=>nil
```