assaf/zombie — Issue #195: Using Zombie for crawling
Author: proxygear
Created: 2011-11-01 12:23:42+00:00
State: open

Hello,

I'm trying different solutions to make web crawling.
I try those few lines :

var zombie = zombie = require("zombie");
zombie.visit("http://client.numericable.fr/", function (err, browser, status) {
// Do something
});

The target is numericable, a french TV provider. It's the typical kind of dirty pages I'll have to browse.
And I already get a lot of errors :

13:14:42 web.1 | error: Dispatching event 'DOMNodeInsertedIntoDocument' failed - More:
13:14:42 web.1 | [object Object]
13:14:42 web.1 | undefined
13:14:42 web.1 | error: Dispatching event 'DOMNodeInsertedIntoDocument' failed - More:
13:14:42 web.1 | [object Object]
13:14:42 web.1 | undefined
13:14:43 web.1 | error: Dispatching event 'DOMNodeInsertedIntoDocument' failed - More:
13:14:43 web.1 | [object Object]
13:14:43 web.1 | undefined
13:14:43 web.1 | error: Dispatching event 'DOMNodeInsertedIntoDocument' failed - More:
13:14:43 web.1 | [object Object]
13:14:43 web.1 | undefined
13:14:43 web.1 | error: Dispatching event 'DOMNodeInsertedIntoDocument' failed - More:
13:14:43 web.1 | [object Object]
13:14:43 web.1 | undefined
13:14:43 web.1 | error: Dispatching event 'DOMNodeInsertedIntoDocument' failed - More:
13:14:43 web.1 | [object Object]
13:14:43 web.1 | undefined
13:14:43 web.1 | error: Dispatching event 'DOMNodeInsertedIntoDocument' failed - More:
13:14:43 web.1 | [object Object]
13:14:43 web.1 | undefined
13:14:43 web.1 | error: Dispatching event 'DOMNodeInsertedIntoDocument' failed - More:
13:14:43 web.1 | [object Object]
13:14:43 web.1 | undefined
13:14:44 web.1 | jQuery is not defined ReferenceError: jQuery is not defined
13:14:44 web.1 | at Object. (/js/colortip-1.0-jquery.js:1:1)
13:14:44 web.1 | in http://client.numericable.fr/
13:14:44 web.1 | jQuery is not defined ReferenceError: jQuery is not defined
13:14:44 web.1 | at Object. (/js/facybox.js:70:2)
13:14:44 web.1 | in http://client.numericable.fr/
13:14:44 web.1 | $ is not defined ReferenceError: $ is not defined
13:14:44 web.1 | at Object. (/js/common.js:1:1)
13:14:44 web.1 | in http://client.numericable.fr/
13:14:44 web.1 | correctionOSL is not defined ReferenceError: correctionOSL is not defined
13:14:44 web.1 | at Object. (/js/s_code.js:54:1)
13:14:44 web.1 | in http://client.numericable.fr/
13:14:44 web.1 | error: Timeout: Cannot read property 'length' of undefined - More:
13:14:44 web.1 | TypeError: Cannot read property 'length' of undefined

For information :
{
"name": "test_zombiiiiii"
, "version": "0.0.1"
, "private": true
, "dependencies": {
"express": "2.5.0"
, "jade": ">= 0.0.1"
, "zombie": ">= 0.1"
, "jsdom" : ">= 0.2.9"
}
}

I would like to know if it's a good idea to use zombie for crawling puprose and if not what other solution I can look for ?

Thanks a lit