Package: chompjs
ChompjsTransforms JavaScript objects into Python data structures.In web scraping, you sometimes need to transform Javascript objects embedded in HTML pages into valid Python dictionaries.chompjsis a library designed to do that as a more powerful replacement of standardjson.loads:>>>chompjs.parse_js_object("{a: 100}"){'a':100}>>>>>>json_lines="""... {'a': 12}... {'b': 13}... {'c': 14}... """>>>forentryinchompjs.parse_js_objects(json_lines):...print(entry)...{'a':12}{'b':13}{'c':14}Reference documentationQuickstart1. installation> pip install chompjsor build from source:$gitclonehttps://github.com/Nykakin/chompjs
$cdchompjs
$pythonsetup.pybuild
$pythonsetup.pyinstallFeaturesThere are two functions available:parse_js_object- try reading first encountered JSON-like object. RaisesValueErroron failureparse_js_objects- returns a generator yielding all encountered JSON-like objects. Can be used to readJSON Lines. Does not raise on invalid input.An example usage withscrapy:importchompjsimportscrapyclassMySpider(scrapy.Spider):# ...defparse(self,response):script_css='script:contains("__NEXT_DATA__")::text'script_pattern=r'__NEXT_DATA__ = (.*);'# warning: for some pages you need to pass replace_entities=True# into re_first to have JSON escaped properlyscript_text=response.css(script_css).re_first(script_pattern)try:json_data=chompjs.parse_js_object(script_text)exceptValueError:self.log('Failed to extract data from{}'.format(response.url))return# work on json_dataParsing ofJSON5 objectsis supported:>>>data="""... {...   // comments...   unquoted: 'and you can quote me on that',...   singleQuotes: 'I can use "double quotes" here',...   lineBreaks: "Look, Mom!\... No\\n's!",...   hexadecimal: 0xdecaf,...   leadingDecimalPoint: .8675309, andTrailing: 8675309.,...   positiveSign: +1,...   trailingComma: 'in objects', andIn: ['arrays',],...   "backwardsCompatible": "with JSON",... }... """>>>chompjs.parse_js_object(data){'unquoted':'and you can quote me on that','singleQuotes':'I can use "double quotes" here','lineBreaks':"Look, Mom! No\n's!",'hexadecimal':912559,'leadingDecimalPoint':0.8675309,'andTrailing':8675309.0,'positiveSign':'+1','trailingComma':'in objects','andIn':['arrays'],'backwardsCompatible':'with JSON'}If the input string is not yet escaped and contains a lot of\\characters, thenunicode_escape=Trueargument might help to sanitize it:>>>chompjs.parse_js_object('{\\\"a\\\": 12}',unicode_escape=True){'a':12}By defaultchompjstries to start with first{or[character it founds, omitting the rest:>>>chompjs.parse_js_object('<div>...</div><script>foo = [1, 2, 3];</script><div>...</div>')[1,2,3]json_paramsargument can be used to pass options to underlyingjson_loads, such asstrictorobject_hook:>>>importdecimal>>>importchompjs>>>chompjs.parse_js_object('[23.2]',json_params={'parse_float':decimal.Decimal})[Decimal('23.2')]RationaleIn web scraping data often is not present directly inside HTML, but instead provided as an embedded JavaScript object that is later used to initialize the page, for example:<html><head>...</head><body>...<scripttype="text/javascript">window.__PRELOADED_STATE__={"foo":"bar"}</script>...</body></html>Standard library functionjson.loadsis usually sufficient to extract this data:>>># scrapy shell file:///tmp/test.html>>>importjson>>>script_text=response.css('script:contains(__PRELOADED_STATE__)::text').re_first('__PRELOADED_STATE__=(.*)')>>>json.loads(script_text){u'foo':u'bar'}The problem is that not all valid JavaScript objects are also valid JSONs. For example all those strings are valid JavaScript objects but not valid JSONs:"{'a': 'b'}"is not a valid JSON because it uses'character to quote'{a: "b"}'is not a valid JSON because property name is not quoted at all'{"a": [1, 2, 3,]}'is not a valid JSON because there is an extra,character at the end of the array'{"a": .99}'is not a valid JSON because float value lacks a leading 0As a result,json.loadsfail to extract any of those:>>>json.loads("{'a': 'b'}")Traceback(mostrecentcalllast):...ValueError:Expectingpropertyname:line1column2(char1)>>>json.loads('{a: "b"}')Traceback(mostrecentcalllast):...ValueError:Expectingpropertyname:line1column2(char1)>>>json.loads('{"a": [1, 2, 3,]}')Traceback(mostrecentcalllast):...ValueError:NoJSONobjectcouldbedecoded>>>json.loads('{"a": .99}')Traceback(mostrecentcalllast):...json.decoder.JSONDecodeError:Expectingvalue:line1column7(char6)chompjslibrary was designed to bypass this limitation, and it allows to scrape such JavaScript objects into proper Python dictionaries:>>>importchompjs>>>>>>chompjs.parse_js_object("{'a': 'b'}"){'a':'b'}>>>chompjs.parse_js_object('{a: "b"}'){'a':'b'}>>>chompjs.parse_js_object('{"a": [1, 2, 3,]}'){'a':[1,2,3]}>>>chompjs.parse_js_object('{"a": .99}'){'a':0.99}Internallychompjsuse a parser written in C to iterate over raw string, fixing its issues along the way. The final result is then passed down to standard library'sjson.loads, ensuring a high speed as compared to full-blown JavaScript parsers such asdemjson.>>>importjson>>>import_chompjs>>>>>>_chompjs.parse('{a: 1}')'{"a":1}'>>>json.loads(_){'a':1}DevelopmentPull requests are welcome.To run unittests$ tox