opened
kmcluckie
username_0
I am using the following Browsershot configuration for the crawler:

```
$browsershot = new Browsershot();
$browsershot->waitForFunction("document.querySelector('#special_element') != null", 500, 5000);
```

Most pages on the site have the `#special_element`, but when the crawler encounters a page that does not have this element it exits with `Symfony\Component\Process\Exception\ProcessFailedException`. This is not caught in the `crawlFailed()` method of the `CrawlObserver`.

The trace provided from the command is as follows:

```
TimeoutError: waiting for function failed: timeout 5000ms exceeded
    at new WaitTask (/home/vagrant/sites/projectX/node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:411:34)
    at DOMWorld.waitForFunction (/home/vagrant/sites/projectX/node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:375:16)
    at Frame.waitForFunction (/home/vagrant/sites/projectX/node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:910:32)
    at Page.waitForFunction (/home/vagrant/sites/projectX/node_modules/puppeteer/lib/cjs/puppeteer/common/Page.js:1328:33)
    at callChrome (/home/vagrant/sites/projectX/vendor/spatie/browsershot/bin/browser.js:235:24)
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  name: 'TimeoutError'
}

  at vendor/spatie/browsershot/src/Browsershot.php:745
    741▕         if ($process->getExitCode() === 2) {
    742▕             throw new ElementNotFound($this->additionalOptions['selector']);
    743▕         }
    744▕
  ➜ 745▕         throw new ProcessFailedException($process);
    746▕     }
    747▕
    748▕     protected function getFullCommand(array $command)
    749▕     {
```

Is this something that the crawler should/could pass to `crawlFailed()` to handle? If not, how can I prevent Browsershot exceptions from halting the process of crawling the rest of the site?
Browsershot exceptions unhandled
issue
created
kaianuar
username_1
I'm having a related problem. I queue 100 urls at a time using an ArrayCrawlQueue. However the whole crawl stops whenever any of the urls encounters a timeout issue.

`The process "PATH=$PATH:/usr/local/bin NODE_PATH=`/usr/bin/node /usr/bin/npm root -g` /usr/bin/node '/home/scraper/public_html/scraper_app/vendor/spatie/browsershot/src/../bin/browser.js' '{"url":"xxx","action":"content","options":{"args":["--no-sandbox"],"viewport":{"width":800,"height":600},"userAgent":"Mozilla\/5.0","dismissDialogs":true,"timeout":30000}}'" exceeded the timeout of 30 seconds.`

I'm not able to catch the error in crawlFailed. Any suggestions on how I can get the script to ignore the error and proceed to the next url in the queue?
comment
created
freekmurze
username_2
I'd accept a PR that catches the error and passes it to `crawlFailed`
comment
created
freekmurze
username_2
Closing for now, still would welcome that PR.
comment
closed
freekmurze
username_2
issue
created
kmcluckie
username_0
Thanks. I had written around the issue for now, but if I come back to that area I'll certainly try to tackle it so that I can provide a PR.
comment
created
MichaelNino
username_3
@username_0 Care to share your resolution? @username_2 Is there a way to pass timeout config from spatie crawler?
comment