Payload in array

When I studied the HTTP Post in the Google Flight, I found the Request Payload is like the following:

[,[[,"aa","[,2,\"S\"]","18563594632275",5]],[,[[,"b_al","no:73"],[,"b_ahr","no:s"],[,"b_am","aa"],[,"b_qu","0"],[,"b_qc","1"]]]]

However, Python does allow None in array, that means, var = [None, 'something'] is right in Python, but var = [ , 'something'] is wrong. What's more, when I post something via lib requests, there is an error :

TypeError: 'NoneType' object is not iterable

So, could you please give me some suggestions of how to deal with the Payload? I searched about it, but in other cases the Payload is a list rather than a array like [, 'something'].


Uh, can I see the body as it went over the wire? Because that payload doesn't look like anything I've ever seen before.


1. For instance,
   https://www.google.fr/flights/#search;f=CFE;t=ORY,CDG;d=2014-04-10;r=2014-04-14.
   When you input the city name (Paris in this example), you will get the
   following.
   2.
2. Request URL:
   https://www.google.fr/flights/rpc
3. Request Method:
   POST
4. Status Code:
   200 OK
5. Request Headers
   1. :host:
      www.google.fr
   2. :method:
      POST
   3. :path:
      /flights/rpc
   4. :scheme:
      https
   5. :version:
      HTTP/1.1
   6. accept:
      _/_
   7. accept-encoding:
      gzip,deflate,sdch
   8. accept-language:
      en-US,en;q=0.8,zh-CN;q=0.6,zh;q=0.4
   9. content-length:
      332
   10. content-type:
       application/json; charset=UTF-8
   11. cookie:
       PREF=ID=bdc670c680013c1d:U=19abbcd0a8a83450:LD=en:TM=1394271727:LM=1394271727:S=QmfR-IjoUHauV5zE;
       NID=67=amO5W8p5BQ4fVLAxx-0BDwEn7HfIMxgmu7qBOUmcDxNIO1kSHXy7zTuzJq4lVTrCL3LeReO6Zt3L6FDexx6A-KkhoN1ssm2yhDhJqCupoP1C03GzgyotwrL4kchf8ZyQ4XDIOyCETdEjgbhutwcyoub7vg5P0xIz-ZMFDeEOaTvHS4TosDwnByJPQJP586F0Lab2HGXMuxgqfVXj-Y1L-vJoE0zYgkoNlVj-SCrKpoFS0Rd7mEIHnjBE58_M;
       HSID=AI16cWqyIltev68Uf; SSID=AlsAFvEjOeGI2t5dK;
       APISID=xY5RyNYIKLf8WiOV/AyRDjN2kPNIj0LPun;
       SAPISID=ciFaHzHJIXPA1jk3/AxCRCO5Y6MZJZWYgC;
       SID=DQAAAGIBAABrDfdaH1R_Yn793rLBvpfcnAKmXxfQrx4-NCpxKVau462gHkkEk-O9efQjJ5it9p5brTq9JvvbmLycn_VQeXz3vYlhCUkSAcE6S-CtnIW9IwSYVG8ymnj14JR4YDfT3E-JgY3sZAyZII4AqD3kAkb0t1SNp75wg_m36xFOayDD3D048PCcd6ziA3XUN7G-9H9cY-TqvK2NAyKxT83ab9pnZ7d_ke1nzbWpmTLq5MCmfE5eSfj31HXHEC_N7djFEeiTojdZG0WZmBgmGU38lYzo2Zsqe_hqQ_lZY5Q_FXTLsAxJBzSd-EiOP1OGGGEpceTOURtJfoAhTz5dl_c-eOVRQoZNrBoJLgY77sgKJQ52060A1vzH-oxmfRKaJFlDCmkl7Oqtlu6VxTbAcq8LaK67FIEftgfAVdg2dQX9o7s6RglTqGdVT31bCr53sSQ3bmlfHjGgIeHS3Lu3bJ3h99UESxi-nM8xnSad9mNWvEnccw;
       S=travel-flights=7TxbQr450yFEt8Fvj_9-mA
   12. origin:
       https://www.google.fr
   13. referer:
       https://www.google.fr/flights/
   14. user-agent:
       Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36
       (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36
   15. x-client-data:
       CPm1yQEIlLbJAQiitskBCKm2yQEIxLbJAQi5iMoBCNiIygE=
   16. x-gwt-cctoken:
       
       ADS25WNRLxnpbQVKxuqSMqiVqxn6s93mt3519cz4gVBbRfMoGZw2GbNb4S87PRNL65-ea7L6gTpcQOZcUjVw9flmw9tBfHmhvCreoXDJoP-2pZZGcp66BFFbMFtQArEPlRAfKzC-wcjQjPA02FnPR7sXYuPhgI4WqyBwtVvMlxQ8kOUozKZMMqfyNWGS4ev15eKrhA
   17. x-gwt-module-base:
       https://www.google.fr/flights/static/
   18. x-gwt-permutation:
       D847595CBA5616E47BFE440BE5D950B6
   19. Request Payload
       1.
       [,[[,"tb","[,[,[[,[\"CFE\"],[\"ORY\",\"CDG\"],\"2014-04-10\"],[,[\"ORY\",\"CDG\"],[\"CFE\"],\"2014-04-14\"]]]]","1846721392154327",11]],[,[[,"b_lr","15:45"],[,"b_al","fs:86"],[,"b_ahr","fs:s"],[,"b_lr","13:207"],[,"b_lr","19:0"],[,"b_lr","13:208"],[,"b_am","tb"],[,"b_pe","4F56A1E74FAE0.AD54F0A.299D"],[,"b_qu","1"],[,"b_qc","1"]]]]
6. Response Headersview source
   1. alternate-protocol:
      443:quic
   2. cache-control:
      no-cache, no-store, max-age=0, must-revalidate
   3. content-encoding:
      gzip
   4. content-length:
      724
   5. content-type:
      application/json; charset=utf-8
   6. date:
      Tue, 25 Mar 2014 08:27:13 GMT
   7. expires:
      Fri, 01 Jan 1990 00:00:00 GMT
   8. pragma:
      no-cache
   9. server:
      GSE
   10. status:
       200 OK
   11. version:
       HTTP/1.1
   12. x-content-type-options:
       nosniff
   13. x-frame-options:
       SAMEORIGIN
   14. x-xss-protection:
       1; mode=block

2014-03-25 8:12 GMT+01:00 Cory Benfield notifications@github.com:

> Uh, can I see the body as it went over the wire? Because that payload
> doesn't look like anything I've ever seen before.
> 
> ## 
> 
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/1974#issuecomment-38536270
> .

## 

Guo Rui
Senior Undergraduate Student
School of Software
Harbin Institute of Technology


So, that request body appears to claim to be JSON, though I'm sure it's not valid JSON as Python fails to parse it. This is not really anything Requests can help you with: you'll need to create that string yourself. You won't even be able to use Python's JSON module to build it as Python has this silly idea that JSON actually needs to match the JSON spec.

Sorry we can't be more helpful, but I strongly suggest you write a very strongly worded letter to Google.fr to tell them to sort their mess out.


I'm not sure how to create a string to do those things...I'm new to Web
tech...Can you show me some examples in such cases? I searched the
documents and didn't find what I want, so I open an issue in Github...

2014-03-25 10:11 GMT+01:00 Cory Benfield notifications@github.com:

> So, that request body appears to claim to be JSON, though I'm sure it's
> not valid JSON as Python fails to parse it. This is not really anything
> Requests can help you with: you'll need to create that string yourself. You
> won't even be able to use Python's JSON module to build it as Python has
> this silly idea that JSON actually needs to match the JSON spec.
> 
> Sorry we can't be more helpful, but I strongly suggest you write a very
> strongly worded letter to Google.fr to tell them to sort their mess out.
> 
> ## 
> 
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/1974#issuecomment-38543308
> .

## 

Guo Rui
Senior Undergraduate Student
School of Software
Harbin Institute of Technology


Unfortunately, there's very limited help I can provide without simply writing your code for you. I'm sorry that this was your first foray into web technologies because Google have screwed you. They've invented their own crazy system that doesn't interoperate with anything. You'll literally need to build a plain string with no particular structure and put your own values into it.


For anyone who ends up looking at this mess in the future, [I wrote an open letter](https://gist.github.com/Lukasa/9758031).


I got it...Perhaps I can try nodeJS to develop the crawler later. Thanks
again!

2014-03-25 10:32 GMT+01:00 Cory Benfield notifications@github.com:

> For anyone who ends up looking at this mess in the future, I wrote an
> open letter https://gist.github.com/Lukasa/9758031.
> 
> ## 
> 
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/1974#issuecomment-38544888
> .

## 

Guo Rui
Senior Undergraduate Student
School of Software
Harbin Institute of Technology