edx/edx-platform — pull request #5431 by bradenmacdonald (2014-09-30 02:21:59+00:00)
Title: Consistently wrap the DataDog API to prevent logged errors & missing stats data

**Background**: While working on #5386, I noticed these warnings in the logs:

```
UnicodeEncodeError: 'ascii' codec can't encode character u'\u03a9' in position 61: ordinal not in range(128)
stevedore.extension: DEBUG: found extension EntryPoint.parse('course-v1 = opaque_keys.edx.locator:CourseLocator')
dd.dogapi: ERROR: couldnt submit statsd point
Traceback (most recent call last):
File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/dogapi/stats/statsd.py", line 29, in add_point
self.socket_sendto(payload, self.address)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u03a9' in position 65: ordinal not in range(128)
```

**Discussion**: Apparently dog_stats_api cannot handle non-ASCII tags. In some parts of the codebase, such as `edx-platform/lms/djangoapps/bulk_email/tasks.py`, all calls to `dog_stats_api` are wrapped with `_statsd_tag()` which prevents these UnicodeErrors. But elsewhere in the codebase, `dog_stats_api` methods called with tags that are unicode strings, potentially causing UnicodeErrors and resulting in statistics being under-reported, should the tags contain any non-ASCII characters.

To fix this, I refactored the code so that *all* calls to the DataDog API are wrapped. At the same time I added a couple other checks/cleanups to the wrapper method to ensure stats reporting is as robust as possible:
* Trim the tags to 200 characters while they are unicode strings, *then* encode them as UTF-8. Doing it vice-versa as the previous `_statsd_tag()` method did can result in splitting multi-byte characters and thus corrupt strings.
* Replace `|` character if present, as statsd uses it as a field separator

I considered making this an upstream patch to the dogapi package but the simplest approach I found would break backwards compatibility, and not doing so would require a larger change (adding the `six` package etc.) as they support Python 2.7 and 3.x. Changing edx-platform is simpler.

**Affects**: Unknkown. Not likely to affect too many courses/instances at this time.

**How to reproduce**: Reverse the one-line fix to student/models.py made in #5386, then run the test that was part of that PR. Check over the detailed test failure log output. You will see the logged warnings quoted above.

**Urgency**: Low. Just a warning in the logs and at worst result in missing stats data.

**Authorship**: This is a contribution from OpenCraft.

**Partner information**: 3rd party-hosted open edX instance, for an edX solutions client (so will need to get some sort of solution merged in)

**Merge deadline**: Mon October 20 (soft requirement to minimize code drift, we may maintain it on the client fork in the meantime)