Repository: getsentry/sentry

Issue:
Add support for "indexed" (or "alternate format") source maps
See: [Source Map Revision 3 Proposal](https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k)

Right now Sentry supports the default source map format, documented in the spec under [Proposed Format](https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k/edit#heading=h.qz3o9nc69um5).

However, the spec also outlines an alternate [Index mapped](https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k/edit#heading=h.535es3xeprgt) format. This format is currently not supported by Sentry, and will be rejected if it is discovered as an artifact when processing an error event's stack trace.

This hasn't been a problem up until now, because AFAIK all popular source-map generating tools produced the default format (e.g. Browserify, Uglify). But it has come to our attention that the **React Native** bundler (as of 0.25.x) will produce this indexed source map format when run with `--dev false`:

``` bash
$ react-native bundle --dev false --entry-file index.ios.js --bundle-output main.bundle --sourcemap-output main.bundle.map
```

Produces the index format:

```
{"version":3,"file":"main.bundle.bundle","sections":[{"offset":{"line":0,"column":0}, ....
```

_Note that when run with --dev true (or --dev is altogether omitted), React Native produces the standard format source map and Sentry ingests this properly._



Discussion:
Here is the [relevant source map building code](https://github.com/facebook/react-native/blob/cccb8db1751572f4ce3800c1d325d15319f573ea/local-cli/bundle/output/unbundle/as-indexed-file.js) (as-indexed.js) from React Native.

So, for this complexity, I'd like to straight up remove it out of Sentry core, for one.

I already have https://github.com/mattrobenolt/python-sourcemap, but it does significantly more, and iirc was heavier in memory usage. Though, I've slimmed it down quite a bit.

But I'd want to support this inside python-sourcemap and then use that instead of shoving more into our core here. I have lots more test cases and unit tests over there.

(We can also move this library under getsentry org if that matters)

@mattrobenolt – makes sense to me. I mean, we may also be looking at having to build a stack trace normalization/formatting lib in Python too (since there is a long term goal of having @getsentry do that rather than browser js libs).

Yeah, I think that's a separate problem, and would probably be worth doing a lib just for that. Then we can try to convert over all the unit tests from js to python.

What's the update on this?

I really can't release an app with the `--dev true` flag and didn't find a workaround..

Another thing I noticed is the comment in the source code after the source map url:

```
//# sourceMappingURL=bundle.ios.map/*ascii:17f584f337b41bd9ff745c4591968d562a2a49d0*/
```

It's `ascii` for non-dev builds and `utf8` for dev builds. I guess it's the encoding of the build but it breaks sentry. It tries to find the file `bundle.ios.map/*ascii:17f584f337b41bd9ff745c4591968d562a2a49d0*/` which obviously doesn't exists.


Resolving patch:
diff --git a/src/sentry/lang/javascript/processor.py b/src/sentry/lang/javascript/processor.py
--- a/src/sentry/lang/javascript/processor.py
+++ b/src/sentry/lang/javascript/processor.py
@@ -34,7 +34,7 @@ class ZeroReturnError(Exception):
 from sentry.utils.strings import truncatechars
 
 from .cache import SourceCache, SourceMapCache
-from .sourcemaps import sourcemap_to_index, find_source
+from .sourcemaps import sourcemap_to_index, find_source, get_inline_content_sources
 
 
 # number of surrounding lines (on each side) to fetch
@@ -758,10 +758,9 @@ def cache_source(self, filename, release):
         sourcemaps.add(sourcemap_url, sourcemap_idx)
 
         # cache any inlined sources
-        for source in sourcemap_idx.sources:
-            next_filename = urljoin(sourcemap_url, source)
-            if source in sourcemap_idx.content:
-                cache.add(next_filename, sourcemap_idx.content[source])
+        inline_sources = get_inline_content_sources(sourcemap_idx, sourcemap_url)
+        for source in inline_sources:
+            self.cache.add(*source)
 
     def populate_source_cache(self, frames, release):
         """
diff --git a/src/sentry/lang/javascript/sourcemaps.py b/src/sentry/lang/javascript/sourcemaps.py
--- a/src/sentry/lang/javascript/sourcemaps.py
+++ b/src/sentry/lang/javascript/sourcemaps.py
@@ -4,6 +4,14 @@
 
 Originally based on https://github.com/martine/python-sourcemap
 
+Sentry implements the Source Map Revision 3 protocol. Specification:
+https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k/edit
+
+Sentry supports both "standard" source maps, and has partial support for "indexed" source
+maps. Specifically, it supports indexed source maps with the "map" section property as
+output by the React Native bundler. It does NOT support indexed source maps with the "url"
+section property.
+
 :copyright: (c) 2010-2014 by the Sentry Team, see AUTHORS for more details.
 :license: BSD, see LICENSE for more details.
 """
@@ -19,6 +27,7 @@
 
 SourceMap = namedtuple('SourceMap', ['dst_line', 'dst_col', 'src', 'src_line', 'src_col', 'name'])
 SourceMapIndex = namedtuple('SourceMapIndex', ['states', 'keys', 'sources', 'content'])
+IndexedSourceMapIndex = namedtuple('IndexedSourceMapIndex', ['offsets', 'maps'])
 
 # Mapping of base64 letter -> integer value.
 B64 = dict(
@@ -110,9 +119,7 @@ def parse_sourcemap(smap):
             yield SourceMap(dst_line, dst_col, src, src_line, src_col, name)
 
 
-def sourcemap_to_index(sourcemap):
-    smap = json.loads(sourcemap)
-
+def _sourcemap_to_index(smap):
     state_list = []
     key_list = []
     src_list = set()
@@ -152,8 +159,73 @@ def sourcemap_to_index(sourcemap):
     return SourceMapIndex(state_list, key_list, src_list, content)
 
 
-def find_source(indexed_sourcemap, lineno, colno):
+def sourcemap_to_index(sourcemap):
+    """
+    Converts a raw sourcemap string to either a SourceMapIndex (basic source map)
+    or IndexedSourceMapIndex (indexed source map w/ "sections")
+    """
+    smap = json.loads(sourcemap)
+
+    if smap.get('sections'):
+        # indexed source map
+        offsets = []
+        maps = []
+        for section in smap.get('sections'):
+            offset = section.get('offset')
+
+            offsets.append((offset.get('line'), offset.get('column')))
+            maps.append(_sourcemap_to_index(section.get('map')))
+
+        return IndexedSourceMapIndex(offsets, maps)
+    else:
+        # standard source map
+        return _sourcemap_to_index(smap)
+
+
+def get_inline_content_sources(sourcemap_index, sourcemap_url):
+    """
+    Returns a list of tuples of (filename, content) for each inline
+    content found in the given source map index. Note that `content`
+    itself is a list of code lines.
+    """
+    out = []
+    if isinstance(sourcemap_index, IndexedSourceMapIndex):
+        for smap in sourcemap_index.maps:
+            out += get_inline_content_sources(smap, sourcemap_url)
+    else:
+        for source in sourcemap_index.sources:
+            next_filename = urljoin(sourcemap_url, source)
+            if source in sourcemap_index.content:
+                out.append((next_filename, sourcemap_index.content[source]))
+    return out
+
+
+def find_source(sourcemap_index, lineno, colno):
+    """
+    Given a SourceMapIndex and a transformed lineno/colno position,
+    return the SourceMap object (which contains original file, line,
+    column, and token name)
+    """
+
     # error says "line no 1, column no 56"
     assert lineno > 0, 'line numbers are 1-indexed'
 
-    return indexed_sourcemap.states[bisect.bisect_right(indexed_sourcemap.keys, (lineno - 1, colno)) - 1]
+    if isinstance(sourcemap_index, IndexedSourceMapIndex):
+        map_index = bisect.bisect_right(sourcemap_index.offsets, (lineno - 1, colno)) - 1
+        offset = sourcemap_index.offsets[map_index]
+        col_offset = 0 if lineno != offset[0] else offset[1]
+        state = find_source(
+            sourcemap_index.maps[map_index],
+            lineno - offset[0],
+            colno - col_offset,
+        )
+        return SourceMap(
+            state.dst_line + offset[0],
+            state.dst_col + col_offset,
+            state.src,
+            state.src_line,
+            state.src_col,
+            state.name
+        )
+    else:
+        return sourcemap_index.states[bisect.bisect_right(sourcemap_index.keys, (lineno - 1, colno)) - 1]


Test patch:
diff --git a/tests/sentry/lang/javascript/fixtures/indexed.min.js b/tests/sentry/lang/javascript/fixtures/indexed.min.js
new file mode 100644
--- /dev/null
+++ b/tests/sentry/lang/javascript/fixtures/indexed.min.js
@@ -0,0 +1,3 @@
+function add(a,b){"use strict";return a+b}
+function multiply(a,b){"use strict";return a*b}function divide(a,b){"use strict";try{return multiply(add(a,b),a,b)/c}catch(e){Raven.captureException(e)}}
+//# sourceMappingURL=indexed.sourcemap.js
diff --git a/tests/sentry/lang/javascript/fixtures/indexed.sourcemap.js b/tests/sentry/lang/javascript/fixtures/indexed.sourcemap.js
new file mode 100644
--- /dev/null
+++ b/tests/sentry/lang/javascript/fixtures/indexed.sourcemap.js
@@ -0,0 +1,32 @@
+{
+    "version": 3,
+    "file": "min.js",
+    "sections": [
+        {
+            "offset": {
+                "line": 0,
+                "column": 0
+            },
+            "map": {
+                "version":3,
+                "sources":["file1.js"],
+                "names":["add","a","b"],
+                "mappings":"AAAA,QAASA,KAAIC,EAAGC,GACf,YACA,OAAOD,GAAIC",
+                "file":"file1.min.js"
+            }
+        },
+        {
+            "offset": {
+                "line": 1,
+                "column": 0
+            },
+            "map": {
+                "version":3,
+                "sources":["file2.js"],
+                "names":["multiply","a","b","divide","add","c","e","Raven","captureException"],
+                "mappings":"AAAA,QAASA,UAASC,EAAGC,GACpB,YACA,OAAOD,GAAIC,EAEZ,QAASC,QAAOF,EAAGC,GAClB,YACA,KACC,MAAOF,UAASI,IAAIH,EAAGC,GAAID,EAAGC,GAAKG,EAClC,MAAOC,GACRC,MAAMC,iBAAiBF",
+                "file":"file2.min.js"
+            }
+        }
+    ]
+}
diff --git a/tests/sentry/lang/javascript/fixtures/unsupported.min.js b/tests/sentry/lang/javascript/fixtures/unsupported.min.js
new file mode 100644
--- /dev/null
+++ b/tests/sentry/lang/javascript/fixtures/unsupported.min.js
@@ -0,0 +1,2 @@
+function add(a,b){"use strict";return a+b}function multiply(a,b){"use strict";return a*b}function divide(a,b){"use strict";try{return multiply(add(a,b),a,b)/c}catch(e){Raven.captureException(e)}}
+//@ sourceMappingURL=unsupported.sourcemap.js
diff --git a/tests/sentry/lang/javascript/fixtures/unsupported.sourcemap.js b/tests/sentry/lang/javascript/fixtures/unsupported.sourcemap.js
new file mode 100644
--- /dev/null
+++ b/tests/sentry/lang/javascript/fixtures/unsupported.sourcemap.js
@@ -0,0 +1,20 @@
+{
+    "version": 3,
+    "file": "min.js",
+    "sections": [
+        {
+            "offset": {
+                "line": 0,
+                "column": 0
+            },
+            "url": "https://example.org/static/1.map"
+        },
+        {
+            "offset": {
+                "line": 1,
+                "column": 0
+            },
+            "url": "https://example.org/static/2.map"
+        }
+    ]
+}
diff --git a/tests/sentry/lang/javascript/test_plugin.py b/tests/sentry/lang/javascript/test_plugin.py
--- a/tests/sentry/lang/javascript/test_plugin.py
+++ b/tests/sentry/lang/javascript/test_plugin.py
@@ -197,6 +197,96 @@ def test_sourcemap_source_expansion(self):
         # its raw and original form should be identical
         assert raw_frame_list[1] == frame_list[1]
 
+    @responses.activate
+    def test_indexed_sourcemap_source_expansion(self):
+        responses.add(responses.GET, 'http://example.com/indexed.min.js',
+                      body=load_fixture('indexed.min.js'))
+        responses.add(responses.GET, 'http://example.com/file1.js',
+                      body=load_fixture('file1.js'))
+        responses.add(responses.GET, 'http://example.com/file2.js',
+                      body=load_fixture('file2.js'))
+        responses.add(responses.GET, 'http://example.com/indexed.sourcemap.js',
+                      body=load_fixture('indexed.sourcemap.js'))
+
+        data = {
+            'message': 'hello',
+            'platform': 'javascript',
+            'sentry.interfaces.Exception': {
+                'values': [{
+                    'type': 'Error',
+                    'stacktrace': {
+                        'frames': [
+                            {
+                                'abs_path': 'http://example.com/indexed.min.js',
+                                'filename': 'indexed.min.js',
+                                'lineno': 1,
+                                'colno': 39,
+                            },
+
+                            {
+                                'abs_path': 'http://example.com/indexed.min.js',
+                                'filename': 'indexed.min.js',
+                                'lineno': 2,
+                                'colno': 44,
+                            },
+
+                        ],
+                    },
+                }],
+            }
+        }
+
+        resp = self._postWithHeader(data)
+        assert resp.status_code, 200
+
+        event = Event.objects.get()
+        assert not event.data['errors']
+
+        exception = event.interfaces['sentry.interfaces.Exception']
+        frame_list = exception.values[0].stacktrace.frames
+
+        frame = frame_list[0]
+        assert frame.pre_context == [
+            'function add(a, b) {',
+            '\t"use strict";',
+        ]
+        assert frame.context_line == '\treturn a + b;'
+        assert frame.post_context == ['}']
+
+        raw_frame_list = exception.values[0].raw_stacktrace.frames
+        raw_frame = raw_frame_list[0]
+        assert raw_frame.pre_context == []
+        assert raw_frame.context_line == 'function add(a,b){"use strict";return a+b}'
+        assert raw_frame.post_context == [
+            'function multiply(a,b){"use strict";return a*b}function divide(a,b){"use strict";try{return multiply(add(a,b),a,b)/c}catch(e){Raven.captureE {snip}',
+            '//# sourceMappingURL=indexed.sourcemap.js',
+            ''
+        ]
+        assert raw_frame.lineno == 1
+
+        frame = frame_list[1]
+        assert frame.pre_context == [
+            'function multiply(a, b) {',
+            '\t"use strict";',
+        ]
+        assert frame.context_line == '\treturn a * b;'
+        assert frame.post_context == [
+            '}',
+            'function divide(a, b) {',
+            '\t"use strict";',
+            '\ttry {',
+            '\t\treturn multiply(add(a, b), a, b) / c;',
+        ]
+
+        raw_frame = raw_frame_list[1]
+        assert raw_frame.pre_context == ['function add(a,b){"use strict";return a+b}']
+        assert raw_frame.context_line == 'function multiply(a,b){"use strict";return a*b}function divide(a,b){"use strict";try{return multiply(add(a,b),a,b)/c}catch(e){Raven.captureE {snip}'
+        assert raw_frame.post_context == [
+            '//# sourceMappingURL=indexed.sourcemap.js',
+            ''
+        ]
+        assert raw_frame.lineno == 2
+
     @responses.activate
     def test_expansion_via_release_artifacts(self):
         project = self.project
@@ -295,3 +385,41 @@ def test_expansion_via_release_artifacts(self):
         ]
         assert frame.context_line == '\treturn a + b;'
         assert frame.post_context == ['}']
+
+    @responses.activate
+    def test_failed_sourcemap_expansion(self):
+        """
+        Tests attempting to parse an indexed source map where each section has a "url"
+        property - this is unsupported and should fail.
+        """
+        responses.add(responses.GET, 'http://example.com/unsupported.min.js',
+                      body=load_fixture('unsupported.min.js'))
+
+        responses.add(responses.GET, 'http://example.com/unsupported.sourcemap.js',
+                      body=load_fixture('unsupported.sourcemap.js'))
+
+        data = {
+            'message': 'hello',
+            'platform': 'javascript',
+            'sentry.interfaces.Exception': {
+                'values': [{
+                    'type': 'Error',
+                    'stacktrace': {
+                        'frames': [
+                            {
+                                'abs_path': 'http://example.com/unsupported.min.js',
+                                'filename': 'indexed.min.js',
+                                'lineno': 1,
+                                'colno': 39,
+                            },
+                        ],
+                    },
+                }],
+            }
+        }
+
+        resp = self._postWithHeader(data)
+        assert resp.status_code, 200
+
+        event = Event.objects.get()
+        assert event.data['errors'] == [{'url': u'http://example.com/unsupported.sourcemap.js', 'type': 'js_invalid_source'}]
diff --git a/tests/sentry/lang/javascript/test_sourcemaps.py b/tests/sentry/lang/javascript/test_sourcemaps.py
--- a/tests/sentry/lang/javascript/test_sourcemaps.py
+++ b/tests/sentry/lang/javascript/test_sourcemaps.py
@@ -3,7 +3,7 @@
 from __future__ import absolute_import
 
 from sentry.lang.javascript.sourcemaps import (
-    SourceMap, parse_vlq, parse_sourcemap, sourcemap_to_index, find_source
+    SourceMap, parse_vlq, parse_sourcemap, sourcemap_to_index, find_source, get_inline_content_sources
 )
 from sentry.testutils import TestCase
 
@@ -19,6 +19,60 @@
     "sourceRoot": "foo"
 }"""
 
+indexed_sourcemap_example = json.dumps({
+    'version': 3,
+    'file': 'min.js',
+    'sections': [
+        {
+            'offset': {
+                'line': 0,
+                'column': 0
+            },
+            'map': {
+                'version': 3,
+                'sources': [
+                    "one.js"
+                ],
+                'sourcesContent': [
+                    ' ONE.foo = function (bar) {\n' +
+                    '   return baz(bar);\n' +
+                    ' };',
+                ],
+                'names': [
+                    "bar",
+                    "baz"
+                ],
+                'mappings': "CAAC,IAAI,IAAM,SAAUA,GAClB,OAAOC,IAAID",
+                'file': "min.js",
+                'sourceRoot': "/the/root"
+            }
+        },
+        {
+            'offset': {
+                'line': 1,
+                'column': 0
+            },
+            'map': {
+                'version': 3,
+                'sources': [
+                    "two.js"
+                ],
+                'sourcesContent': [
+                    ' TWO.inc = function (n) {\n' +
+                    '   return n + 1;\n' +
+                    ' };'
+                ],
+                'names': [
+                    "n"
+                ],
+                'mappings': "CAAC,IAAI,IAAM,SAAUA,GAClB,OAAOA",
+                'file': "min.js",
+                'sourceRoot': "/the/root"
+            }
+        }
+    ]
+})
+
 
 class ParseVlqTest(TestCase):
     def test_simple(self):
@@ -53,6 +107,24 @@ def test_simple(self):
         assert result == SourceMap(dst_line=0, dst_col=191, src='foo/file2.js', src_line=9, src_col=25, name='e')
 
 
+class GetInlineContentSourcesTest(TestCase):
+    def test_no_inline(self):
+        # basic sourcemap fixture has no inlined sources, so expect an empty list
+        indexed_sourcemap = sourcemap_to_index(sourcemap)
+
+        sources = get_inline_content_sources(indexed_sourcemap, 'https://example.com/static/')
+        assert sources == []
+
+    def test_indexed_inline(self):
+        indexed_sourcemap = sourcemap_to_index(indexed_sourcemap_example)
+
+        sources = get_inline_content_sources(indexed_sourcemap, 'https://example.com/static/')
+        assert sources == [
+            ('https://example.com/the/root/one.js', [' ONE.foo = function (bar) {', '   return baz(bar);', ' };']),
+            ('https://example.com/the/root/two.js', [' TWO.inc = function (n) {', '   return n + 1;', ' };'])
+        ]
+
+
 class ParseSourcemapTest(TestCase):
     def test_basic(self):
         smap = json.loads(sourcemap)
@@ -95,3 +167,39 @@ def test_basic(self):
             SourceMap(dst_line=0, dst_col=174, src='foo/file2.js', src_line=9, src_col=8, name='captureException'),
             SourceMap(dst_line=0, dst_col=191, src='foo/file2.js', src_line=9, src_col=25, name='e'),
         ]
+
+
+class ParseIndexedSourcemapTest(TestCase):
+    # Tests lookups that fall exactly on source map token boundaries
+    # https://github.com/mozilla/source-map/blob/master/test/test-source-map-consumer.js#138
+    def test_exact_mappings(self):
+        indexed_sourcemap = sourcemap_to_index(indexed_sourcemap_example)
+
+        # one.js
+        assert find_source(indexed_sourcemap, 1, 1) == \
+            SourceMap(dst_line=0, dst_col=1, src='/the/root/one.js', src_line=0, src_col=1, name=None)
+        assert find_source(indexed_sourcemap, 1, 18) == \
+            SourceMap(dst_line=0, dst_col=18, src='/the/root/one.js', src_line=0, src_col=21, name='bar')
+        assert find_source(indexed_sourcemap, 1, 28) == \
+            SourceMap(dst_line=0, dst_col=28, src='/the/root/one.js', src_line=1, src_col=10, name='baz')
+
+        # two.js
+        assert find_source(indexed_sourcemap, 2, 18) == \
+            SourceMap(dst_line=1, dst_col=18, src='/the/root/two.js', src_line=0, src_col=21, name='n')
+        assert find_source(indexed_sourcemap, 2, 21) == \
+            SourceMap(dst_line=1, dst_col=21, src='/the/root/two.js', src_line=1, src_col=3, name=None)
+        assert find_source(indexed_sourcemap, 2, 21) == \
+            SourceMap(dst_line=1, dst_col=21, src='/the/root/two.js', src_line=1, src_col=3, name=None)
+
+    # Tests lookups that fall inside source map token boundaries
+    # https://github.com/mozilla/source-map/blob/master/test/test-source-map-consumer.js#181
+    def test_fuzzy_mapping(self):
+        indexed_sourcemap = sourcemap_to_index(indexed_sourcemap_example)
+
+        # one.js
+        assert find_source(indexed_sourcemap, 1, 20) == \
+            SourceMap(dst_line=0, dst_col=18, src='/the/root/one.js', src_line=0, src_col=21, name='bar')
+        assert find_source(indexed_sourcemap, 1, 30) == \
+            SourceMap(dst_line=0, dst_col=28, src='/the/root/one.js', src_line=1, src_col=10, name='baz')
+        assert find_source(indexed_sourcemap, 2, 12) == \
+            SourceMap(dst_line=1, dst_col=9, src='/the/root/two.js', src_line=0, src_col=11, name=None)