mozilla/marketplace-tests-gaia — pull request #10 by bobsilverberg (2014-01-09 01:02:32+00:00)
Title: Override the launch method in GaiaApps to ignore pesky Javascript errors

There have been a lot of errors on `b2g.hamachi.mozilla-aurora.marketplace`, most of which have been a JavascriptException that is thrown during the launch of the Marketplace app, which Marionette then catches and causes the launch to fail. This is a non-critical JS exception, but it causes the test to fail.

A sample stack trace is:

```
test_marketplace_feedback_user (test_marketplace_feedback_login.TestMarketplaceFeedback) ... ERROR

======================================================================
ERROR: None
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/jenkins/workspace/b2g.hamachi.mozilla-aurora.marketplace/.env/local/lib/python2.7/site-packages/marionette_client-0.7.1-py2.7.egg/marionette/marionette_test.py", line 143, in run
testMethod()
File "/var/jenkins/workspace/b2g.hamachi.mozilla-aurora.marketplace/marketplacetests/tests/test_marketplace_feedback_login.py", line 24, in test_marketplace_feedback_user
self.marketplace.launch()
File "/var/jenkins/workspace/b2g.hamachi.mozilla-aurora.marketplace/marketplacetests/marketplace/app.py", line 48, in launch
Base.launch(self, launch_timeout=120000)
File "/var/jenkins/workspace/b2g.hamachi.mozilla-aurora.marketplace/.env/local/lib/python2.7/site-packages/gaiatest-0.21.2-py2.7.egg/gaiatest/apps/base.py", line 26, in launch
self.app = self.apps.launch(self.name, launch_timeout=launch_timeout)
File "/var/jenkins/workspace/b2g.hamachi.mozilla-aurora.marketplace/.env/local/lib/python2.7/site-packages/gaiatest-0.21.2-py2.7.egg/gaiatest/gaia_test.py", line 75, in launch
result = self.marionette.execute_async_script("GaiaApps.launchWithName('%s')" % name, script_timeout=launch_timeout)
File "/var/jenkins/workspace/b2g.hamachi.mozilla-aurora.marketplace/.env/local/lib/python2.7/site-packages/marionette_client-0.7.1-py2.7.egg/marionette/marionette.py", line 1073, in execute_async_script
filename=os.path.basename(frame[0]))
File "/var/jenkins/workspace/b2g.hamachi.mozilla-aurora.marketplace/.env/local/lib/python2.7/site-packages/marionette_client-0.7.1-py2.7.egg/marionette/marionette.py", line 577, in _send_message
self._handle_error(response)
File "/var/jenkins/workspace/b2g.hamachi.mozilla-aurora.marketplace/.env/local/lib/python2.7/site-packages/marionette_client-0.7.1-py2.7.egg/marionette/marionette.py", line 612, in _handle_error
raise JavascriptException(message=message, status=status, stacktrace=stacktrace)
JavascriptException: TypeError: widgetFrame is null
stacktrace:
@app://system.gaiamobile.org/js/cost_control.js, line 70
TEST-UNEXPECTED-FAIL | test_marketplace_feedback_login.py test_marketplace_feedback_login.TestMarketplaceFeedback.test_marketplace_feedback_user |
----------------------------------------------------------------------
Ran 1 test in 74.436s

FAILED (errors=1)
```

We saw something similar in the gaia-ui-tests job awhile back and had to wait for a developer to address it.

Because this repo is small and focussed on testing Marketplace functionality, as opposed to general gaia reliability, it was suggested that maybe we should just ignore these errors. I believe that Bebe proposed a similar approach at the time and it wasn't favoured, but I discussed this with Zac on IRC and we thought it might be appropriate for this repo.

Hence, I bring you this pull request. It is a bit ugly as I had to override the launch method in `GaiaApps`, but the plus side is that, if we do decide to accept this patch, we now have a template going forward for any other times we need to customize the behaviour of gaiatest for this repo.

I'm not convinced that this is the best approach, and I'm going to try something simpler as well, and probably submit that as a second PR, so I am interested in people's feedback on this.

Pinging @zacc @davehunt @bebef1987 for feedback.