Repository: dr-leo/pandaSDMX
Pull request #51 (2017-04-04 23:36:26)

SDMXJSON tests & fixes
Adds `tests/tests_json.py`, which takes test coverage to ~80% for SDMXJSON.

In order to get the tests to pass for `tests/data/json/exr-flat.json`, I had to rewrite `iter_generic_obs`. I did this with reference to `iter_generic_series_obs` and minimal changes.

Two possible things I could see needing work before this can be merged:
- Use one private method for both `iter_generic_obs` and `iter_generic_series_obs`, since the code is very similar, and make the actual methods thin wrappers.
- `tests/data/json/exr-flat.json` seems to be loaded without the extra attributes (FREQ, CURRENCY_DENOM) etc. as MultiIndex levels—maybe this is because it's loaded as a `pd.Series`, and those are typically returned as levels on a columns `pd.MultiIndex` of a `pd.DataFrame`? If this should be handled some other way, happy to make the required changes.


Patch:
diff --git a/.gitignore b/.gitignore
index 7a7e68f..a71352b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,2 +1,4 @@
+*.cache
+*.egg-info
 .coverage
 __pycache__
diff --git a/doc/agencies.rst b/doc/agencies.rst
index ca9dfe2..9049097 100644
--- a/doc/agencies.rst
+++ b/doc/agencies.rst
@@ -1,10 +1,38 @@
 .. _agencies:    
     
-Dealign with agencies (data providers)
+Data providers
 =========================================
 
+
+Overview
+-----------
 Overview
 ----------------------------------
 
+pandaSDMX supports a number of data providers out of the box. Each data provider
+is configured by an item in ``agencies.json`` in the package root. Data providers are
+identified by an uppercased string such as "ECB" or "OECD". For each pre-configured data provider, ``agencies.json`` contains
+the URL and name of the SDMX API and potentially some additional
+metadata about the provider's API.
+
+Other data providers can be configured by passing a suitable json-file to the
+:meth:`pandasdmx.api.Request.add_agency`.
+  
+
+Pre-configured data providers
+-----------------------------------
+
+This section describes the data providers supported
+out of the box.
+
+European Central Bank (ECB)
+:::::::::::::::::::
+
+
+European Central Bank (ECB)
+::::::::::::::::::::::::::::
+
+  
+
 tbc
 
diff --git a/doc/index.rst b/doc/index.rst
index 72c638c..fd61b48 100644
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -3,13 +3,31 @@ pandaSDMX: Statistical Data and Metadata eXchange in Python
 
 
 pandaSDMX is an Apache 2.0-licensed `Python <http://www.python.org>`_ 
-package aimed at becoming the 
-most intuitive and versatile tool to retrieve and acquire statistical data and metadata
-disseminated in `SDMX <http://www.sdmx.org>`_ format. 
-It supports out of the box 
-the SDMX services of the European statistics office (Eurostat), 
-the European Central Bank (ECB), the French National Institute for statistics (INSEE), 
-the Australian Bureau of Statistics, and the OECD (JSON only). 
+library to retrieve and acquire statistical data
+and metadata disseminated in
+`SDMX <http://www.sdmx.org>`_ 2.1, an ISO-standard
+widely used by institutions
+such as statistics offices, central banks, and international organisations.
+pandaSDMX exposes datasets and related structural metadata 
+including dataflows, code-lists, 
+and datastructure definitions as pandas Series 
+or multi-indexed DataFrames. 
+
+As of May 2017, pandaSDMX has built-in support for the following agencies (others may be  
+configured by the user): 
+
+* `Australian Bureau of Statistics (ABS) <http://www.abs.gov.au/>`_ 
+* `European Central Bank (ECB) <http://www.ecb.europa.eu/stats/ecb_statistics/co-operation_and_standards/sdmx/html/index.en.html>`_
+* `Eurostat <http://ec.europa.eu/eurostat/web/sdmx-web-services/rest-sdmx-2.1>`_
+* `French National Institute for Statistics (INSEE) 
+  <http://www.bdm.insee.fr/bdm2/statique?page=sdmx>`_
+* `International Monetary Fund (IMF) - SDMX Central only 
+  <https://sdmxcentral.imf.org/>`_   
+* `Organisation for Economic Cooperation and Development (OECD)
+  <http://stats.oecd.org/SDMX-JSON/>`_  
+* `United Nations Statistics Division (UNSD) <https://unstats.un.org/home/>`_
+* `UNESCO (free registration required) <https://apiportal.uis.unesco.org/getting-started>`_ 
+ 
 pandaSDMX can export data and metadata as `pandas <http://pandas.pydata.org>`_ DataFrames, 
 the gold-standard 
 of data analysis in Python. 
@@ -59,10 +77,10 @@ concept schemes, and corresponding code lists.
     estat = Request('ESTAT')
     # Download the metadata and expose it as a dict mapping resource names to pandas DataFrames
     metadata = estat.datastructure('DSD_une_rt_a').write()
-    # Show some code lists
-    metadata.codelist.ix[['AGE', 'UNIT']]
+    # Sort the codelist DataFrame by index and show some code lists
+    metadata.codelist.sort_index().loc[['AGE', 'UNIT']]
     
-Next we download a data set. We use codes from the code list 'GEO'
+Next we download a dataset. We use codes from the code list 'GEO'
 to obtain data on Greece, Ireland and Spain only.
 
 .. ipython:: python
diff --git a/doc/requirements.txt b/doc/requirements.txt
new file mode 100644
index 0000000..efdad3e
--- /dev/null
+++ b/doc/requirements.txt
@@ -0,0 +1,8 @@
+ipython>=4.0.0
+jsonpath-rw>=1.4.0
+lxml>=3.3.0
+pandas>=0.18.0
+pytest>=2.8.0
+requests>=2.7
+setuptools>19.0
+Sphinx>=1.4.1
diff --git a/pandasdmx/agencies.json b/pandasdmx/agencies.json
index 0b8d61e..b49fa73 100644
--- a/pandasdmx/agencies.json
+++ b/pandasdmx/agencies.json
@@ -1,26 +1,42 @@
 {
     "": null,
     "ESTAT": {
+        "id": "ESTAT",
         "resources": {},
         "url": "http://ec.europa.eu/eurostat/SDMX/diss-web/rest",
         "name": "Eurostat"
     },
     "UNSD": {
+        "id": "UNSD",
+        "supports_series_keys_only": true,
         "resources": {},
         "url": "http://data.un.org/WS/rest",
         "name": "United Nations Statistics Division"
-    },    
+    },
+    "UNESCO": {
+        "id": "UNESCO",
+        "supports_series_keys_only": true,
+        "headers": {
+            "accept": "application/vnd.sdmx.structure+xml;version=2.1"
+        },
+        "resources": {},
+        "url": "http://api.uis.unesco.org/sdmx",
+        "name": "UNESCO (subscription key required)"
+    },
     "IMF_SDMXCENTRAL": {
+        "id": "IMF",
         "resources": {},
         "url": "https://sdmxcentral.imf.org/ws/public/sdmxapi/rest",
         "name": "International Monetary Fund - SDMXCentral"
     },    
     "SGR": {
+        "id": "SGR",
         "resources": {},
         "url": "https://registry.sdmx.org/ws/rest",
         "name": "SDMX Global Registry"
     },
     "INSEE": {
+        "id": "INSEE",
         "resources": {
             "data": {
                 "headers": {
@@ -32,6 +48,7 @@
         "name": "Institut national de la statistique et des \u00e9tudes \u00e9conomiques"
     },
     "OECD": {
+        "id": "OECD",
         "resources": {
             "data": {
                 "headers": {},
@@ -42,6 +59,7 @@
         "name": "Organisation for Economic Co-operation and Development"
     },
     "ECB": {
+        "id": "ECB",
         "supports_series_keys_only": true,
         "resources": {
             "data": {
@@ -52,6 +70,7 @@
         "name": "European Central Bank"
     },
     "ABS": {
+        "id": "ABS",
         "resources": {
             "data": {
                 "headers": {},
diff --git a/pandasdmx/api.py b/pandasdmx/api.py
index 5640e0a..520179f 100644
--- a/pandasdmx/api.py
+++ b/pandasdmx/api.py
@@ -16,9 +16,8 @@ SOAP interface.
 
 '''
 
-import pandasdmx
 from pandasdmx import remote
-from pandasdmx.utils import str_type
+from pandasdmx.utils import str_type, namedtuple_factory
 import pandas as PD
 from pkg_resources import resource_string
 from importlib import import_module
@@ -166,7 +165,8 @@ class Request(object):
                 self.cache[cache_id] = df
             return df
 
-    def get(self, resource_type='', resource_id='', agency='', key='',
+    def get(self, resource_type='', resource_id='', agency='',
+            version='latest', key='',
             params=None, headers={},
             fromfile=None, tofile=None, url=None, get_footer_url=(30, 3),
             memcache=None, writer=None):
@@ -248,16 +248,14 @@ class Request(object):
         else:
             # Construct URL from args unless ``tofile`` is given
             # Validate args
-            if params is None:
-                params = {}
-            if not agency:
-                agency = self.agency
+            params = params or {}
+            agency = agency or self.agency
             # Validate resource if no filename is specified
             if not fromfile and resource_type not in self._resources:
                 raise ValueError(
                     'resource must be one of {0}'.format(self._resources))
             # resource_id: if it is not a str or unicode type,
-            # but, e.g., a invalid DataflowDefinition,
+            # but, e.g., an invalid Dataflow Definition,
             # extract its ID
             if resource_id and not isinstance(resource_id, (str_type, str)):
                 resource_id = resource_id.id
@@ -280,6 +278,7 @@ class Request(object):
                     'resources'].get(resource_type)
                 if resource_cfg:
                     headers = resource_cfg.get('headers') or {}
+
             # Construct URL from the given non-empty substrings.
             # if data is requested, omit the agency part. See the query
             # examples
@@ -289,7 +288,10 @@ class Request(object):
             # Any parameters are appended by remote module.
             if self.agency:
                 parts = [self._agencies[self.agency]['url'],
-                         resource_type, agency, resource_id, key]
+                         resource_type,
+                         self._agencies[self.agency].get(
+                             'id', agency),  # agency ID
+                         resource_id, version, key]
                 base_url = '/'.join(filter(None, parts))
 
                 # Set references to sensible defaults
@@ -480,25 +482,44 @@ class Request(object):
         parts = [key_l.get(name, '') for name in dim_names]
         return '.'.join(parts)
 
-    def data_size(self, flow_id, key=None, detail=0):
+    def preview_data(self, flow_id, key=None, count=True, total=True):
         '''
-        Calculates number of series for a prospective data query allowing for
-        keys with multiple dimension values.
-        It uses the complete list of series keys for a dataflow rather than constraints and DSD.
+        Get keys or number of series for a prospective dataset query allowing for
+        keys with multiple values per dimension.
+        It downloads the complete list of series keys for a dataflow rather than using constraints and DSD. This feature is,
+        however, not supported by all data providers.
+        ECB and UNSD are known to work.
 
         Args:
 
         flow_id(str): dataflow id
-        key(dict): key mapping dimension names to values or lists of values.
-            Must have been validated before. It is not checked if key values
-            are actually valid dimension names and values.
 
-            detail(int): determins the return type and amount of information 
-                0: returns number of series per key combination
-                1: returns boolean series as indexers per key combination
-                2: returns single DataFrame with all matching series keys
+        key(dict): optional key mapping dimension names to values or lists of values.
+            Must have been validated before. It is not checked if key values
+            are actually valid dimension names and values. Default: {}
+
+        count(bool): if True (default), return the number of series
+            of the dataset designated by flow_id and key. If False,
+            the actual keys are returned as a pandas DataFrame or dict of dataframes, depending on
+the value of 'total'.
+
+        total(bool): if True (default), return the aggregate number
+            of series or a single dataframe (depending on the value of 'count'). If False,
+            return a dict mapping keys to dataframes of series keys.
+            E.g., if key={'COUNTRY':'IT+CA+AU'}, the dict will
+            have 3 items describing the series keys for each country
+            respectively. If 'count' is True, dict values will be int rather than
+            PD.DataFrame.
         '''
         all_keys = self.series_keys(flow_id)
+        # Handle the special case that no key is provided
+        if not key:
+            if count:
+                return all_keys.shape[0]
+            else:
+                return all_keys
+
+        # So there is a key specifying at least one dimension value.
         # Wrap single values in 1-elem list for uniform treatment
         key_l = {k: [v] if isinstance(v, str_type) else v
                  for k, v in key.items()}
@@ -506,26 +527,31 @@ class Request(object):
         dim_names = [k for k in all_keys if k in key]
         # Drop columns that are not in the key
         key_df = all_keys.loc[:, dim_names]
-        if detail == 2:
+        if total:
             # DataFrame with matching series keys
             bool_series = reduce(
                 and_, (key_df.isin(key_l)[col] for col in dim_names))
-            return all_keys[bool_series]
-        elif detail in [0, 1]:
+            if count:
+                return bool_series.value_counts()[True]
+            else:
+                return all_keys[bool_series]
+        else:
             # Dict of value combinations as dict keys
             key_product = product(*(key_l[k] for k in dim_names))
-            matches = {k: reduce(and_, (key_df.isin({k1: [v1]
-                                                     for k1, v1 in zip(dim_names, k)})[col]
-                                        for col in dim_names))
+            # Replace key tuples by namedtuples
+            PartialKey = namedtuple_factory('PartialKey', dim_names)
+
+            matches = {PartialKey(k): reduce(and_, (key_df.isin({k1: [v1]
+                                                                 for k1, v1 in zip(dim_names, k)})[col]
+                                                    for col in dim_names))
                        for k in key_product}
-            if detail == 1:
-                # dict with boolean series as values for each combination
-                return matches
+
+            if not count:
+                # dict mapping each key to DataFrame with selected key-set
+                return {k: all_keys[v] for k, v in matches.items()}
             else:
                 # Number of series per key
                 return {k: v.value_counts()[True] for k, v in matches.items()}
-        else:
-            raise ValueError('`detail`must be 0, 1 or 2')
 
 
 class Response(object):
diff --git a/pandasdmx/model.py b/pandasdmx/model.py
index 1f96c9d..bc28e20 100644
--- a/pandasdmx/model.py
+++ b/pandasdmx/model.py
@@ -324,7 +324,7 @@ class Concept(Item):
 
 class Component(IdentifiableArtefact):
 
-    @property  # not working yet
+    @property
     def concept_identity(self):
         return self._reader.read_instance(Ref, self, offset='concept_identity')
 
@@ -502,7 +502,7 @@ class DataAttribute(Component):
         return self._reader.read_as_str('assignment_status', self)
 
 
-class DataStructureDefinition(Structure):
+class DataStructureDefinition(Structure, Constrainable):
 
     def __init__(self, *args, **kwargs):
         super(DataStructureDefinition, self).__init__(*args, **kwargs)
diff --git a/pandasdmx/reader/sdmxjson.py b/pandasdmx/reader/sdmxjson.py
index a11a4bd..e59effd 100644
--- a/pandasdmx/reader/sdmxjson.py
+++ b/pandasdmx/reader/sdmxjson.py
@@ -55,6 +55,25 @@ class Reader(BaseReader):
         self._dataset_dim = d.get('dataSet', [])
         self._series_dim = d['series']
         self._obs_dim = d['observation']
+        self._dataset_dim_key = {dim['keyPosition']: dim['id']
+                                 for dim in self._dataset_dim}
+        self._dataset_dim_values = {dim['keyPosition']: dim['values'][0]['id']
+                                    for dim in self._dataset_dim}
+        if self._series_dim:
+            self._key_len = len(self._dataset_dim) + len(self._series_dim)
+            # Map keyPositions of dimensions at series level to dimension IDs, like with dataset-level dims above.
+            # In case of cross-sectional dataset, the only dimension at series level has no
+            # keyPosition, eg. TIME_PERIOD. Instead, the keyPosition of the dim at observation
+            # is used to fill the gap.
+            self._series_dim_key = {dim.get('keyPosition',
+                                            self._obs_dim[0].get('keyPosition')):
+                                    dim['id'] for dim in self._series_dim}
+            self.SeriesKeyTuple = namedtuple_factory('SeriesKeyTuple',
+                                                     (self._dataset_dim_key.get(i) or self._series_dim_key.get(i)
+                                                      for i in range(self._key_len)))
+        else:
+            # Dataset must be flat
+            self._key_len = len(self._dataset_dim) + len(self._obs_dim)
         self.obs_attr_id = [d['id'] for d in self._obs_attrib]
         # init message instance
         cls = model.DataMessage
@@ -68,7 +87,8 @@ class Reader(BaseReader):
         '''
         Save source to file by calling `write` on the root element.
         '''
-        return json.dumps(self.message._elem, filename)
+        with open(filename, 'w') as fp:
+            return json.dump(self.message._elem, fp, indent=4, sort_keys=True)
 
     _paths = {
         #         'footer_text': 'com:Text/text()',
@@ -84,9 +104,9 @@ class Reader(BaseReader):
         #         'agencyID': '@agencyID',
         #         'maintainable_parent_id': '@maintainableParentID',
         #         'value': 'com:Value/text()',
-        'headerID': 'id',
-        #         'header_prepared': 'mes:Prepared/text()',
-        #         'header_sender': 'mes:Sender/@*',
+        'headerID': '$.header.id',
+        'header_prepared': '$.header.prepared',
+        'header_sender': '$.header.sender.*',
         #         'header_receiver': 'mes:Receiver/@*',
         #         'assignment_status': '@assignmentStatus',
         #         'error': 'mes:error/@*',
@@ -134,11 +154,11 @@ class Reader(BaseReader):
         #         model.CubeRegion: 'str:CubeRegion',
         #         model.KeyValue: 'com:KeyValue',
         #         model.Ref: 'Ref',
-        model.Header: 'header',
+        model.Header: '$.header',
         #         model.Annotation: 'com:Annotations/com:Annotation',
         #         model.Group: 'gen:Group',
         #         model.Series: 'gen:Series',
-        model.DataSet: 'dataSets[0]',
+        model.DataSet: '$.dataSets[0]',
         #         'int_str_names': './*[local-name() = $name]/@xml:lang',
         #         model.Representation: 'str:LocalRepresentation',
         #         'int_str_values': './*[local-name() = $name]/text()',
@@ -188,32 +208,53 @@ class Reader(BaseReader):
     _SeriesObsTuple = namedtuple_factory(
         'SeriesObservation', ('dim', 'value', 'attrib'))
 
+    # Operators
+    getitem0 = itemgetter(0)
+    getitem_key = itemgetter('_key')
+
     def iter_generic_obs(self, sdmxobj, with_value, with_attributes):
-        for obs in self._paths['generic_obs_path'](sdmxobj._elem):
-            # Construct the namedtuple for the ObsKey.
-            # The namedtuple class is created on first iteration.
-            obs_key_values = self._paths['obs_key_values_path'](obs)
-            try:
-                obs_key = ObsKeyTuple._make(obs_key_values)
-            except NameError:
-                obs_key_id = self._paths['obs_key_id_path'](obs)
-                ObsKeyTuple = namedtuple_factory('ObsKey', obs_key_id)
-                obs_key = ObsKeyTuple._make(obs_key_values)
-            if with_value:
-                obs_value = self._paths['obs_value_path'](obs)[0]
-            else:
-                obs_value = None
-            if with_attributes:
-                obs_attr_values = self._paths['attr_values_path'](obs)
-                obs_attr_id = self._paths['attr_id_path'](obs)
-                obs_attr_type = namedtuple_factory(
-                    'ObsAttributes', obs_attr_id)
-                obs_attr = obs_attr_type(*obs_attr_values)
+        # Make type namedtuple for obs_key. It must be
+        # merged with any dimension values at dataset level maintaining the
+        # key position order.
+        # Note that the measure dimension (such as TIME_PERIOD) has no key position.
+        # We fill this gap by injecting the highest key position.
+        _obs_dim_key = {dim.get('keyPosition', self._key_len - 1): dim['id']
+                        for dim in self._obs_dim}
+        _GenericObsKey = namedtuple_factory('GenericObservationKey',
+                                            (self._dataset_dim_key.get(d)
+                                             or _obs_dim_key.get(d)
+                                             for d in range(self._key_len)))
+        obs_l = sorted(sdmxobj._elem.value['observations'].items(),
+                       key=self.getitem0)
+        for dim, value in obs_l:
+            # Construct the key for this observation
+            key_idx = [int(i) for i in dim.split(':')]
+            obs_key_values = [d['values'][i]['id'] for i, d in
+                              zip(key_idx, self._obs_dim)]
+            obs_key = _GenericObsKey._make(self._dataset_dim_values.get(d)
+                                           or obs_key_values.pop(0)
+                                           for d in range(self._key_len))
+
+            # Read the value
+            obs_value = value[0] if with_value else None
+
+            # Read any attributes
+            if with_attributes and len(value) > 1:
+                obs_attr_idx = value[1:]
+                obs_attr_raw = [(d['id'],
+                                 d['values'][i].get('id',
+                                                    d['values'][i]['name']))
+                                for i, d in zip(obs_attr_idx, self._obs_attrib)]
+                if obs_attr_raw:
+                    obs_attr_id, obs_attr_values = zip(*obs_attr_raw)
+                    obs_attr_type = namedtuple_factory(
+                        'ObsAttributes', obs_attr_id)
+                    obs_attr = obs_attr_type(*obs_attr_values)
+                else:
+                    obs_attr = None
             else:
                 obs_attr = None
-            yield self._ObsTuple(obs_key, obs_value, obs_attr)
-
-    getitem_key = itemgetter('_key')
+            yield self._SeriesObsTuple(obs_key, obs_value, obs_attr)
 
     def generic_series(self, sdmxobj):
         for key, series in sdmxobj._elem.value['series'].items():
@@ -225,17 +266,13 @@ class Reader(BaseReader):
         return []
 
     def series_key(self, sdmxobj):
-        # pull down dataset key
-        full_key_ids = [d['id'] for d in self._dataset_dim]
-        full_key_values = [d['values'][0]['id'] for d in self._dataset_dim]
         key_idx = [int(i) for i in sdmxobj._elem['_key'].split(':')]
-        series_key_ids = [d['id'] for d in self._series_dim]
         series_key_values = [d['values'][i]['id'] for i, d in
                              zip(key_idx, self._series_dim)]
-        full_key_ids.extend(series_key_ids)
-        full_key_values.extend(series_key_values)
-        SeriesKeyTuple = namedtuple_factory('SeriesKey', full_key_ids)
-        return SeriesKeyTuple._make(full_key_values)
+        full_key_values = [self._dataset_dim_values.get(d)
+                           or series_key_values.pop(0)
+                           for d in range(self._key_len)]
+        return self.SeriesKeyTuple._make(full_key_values)
 
     def group_key(self, sdmxobj):
         group_key_id = self._paths['group_key_id_path'](sdmxobj._elem)
@@ -262,8 +299,6 @@ class Reader(BaseReader):
             attrib_ids, attrib_values = zip(*attrib_list)
             return namedtuple_factory('Attrib', attrib_ids)(*attrib_values)
 
-    getitem0 = itemgetter(0)
-
     def iter_generic_series_obs(self, sdmxobj, with_value, with_attributes,
                                 reverse_obs=False):
         obs_l = sorted(sdmxobj._elem['observations'].items(),
diff --git a/pandasdmx/reader/sdmxml.py b/pandasdmx/reader/sdmxml.py
index 4aaaa64..71d7324 100644
--- a/pandasdmx/reader/sdmxml.py
+++ b/pandasdmx/reader/sdmxml.py
@@ -76,7 +76,7 @@ class Reader(BaseReader):
         'assignment_status': '@assignmentStatus',
         'error': 'mes:error/@*',
         'ref_version': '@version',
-        'concept_id': 'str:ConceptIdentity',
+        'concept_identity': 'str:ConceptIdentity',
         'position': '@position',
         'isfinal': '@isfinal',
         'ref_package': '@package',
diff --git a/pandasdmx/writer/structure2pd.py b/pandasdmx/writer/structure2pd.py
index 7906886..82dd8b8 100644
--- a/pandasdmx/writer/structure2pd.py
+++ b/pandasdmx/writer/structure2pd.py
@@ -81,25 +81,20 @@ class Writer(BaseWriter):
                         columns=['name'], lang='en'):
 
         def make_column(scheme, item):
-            if rows == 'codelist' and dsd:
-                if scheme is item:
-                    item = scheme[1]
-                scheme = scheme[1]
-            if scheme is item:
-                raw = [getattr(scheme, s) for s in columns]
-            else:
-                # Prepare for flat list where item is None and all the
-                # gist is in scheme.
-                if item is None:
-                    item = scheme
-                raw = [getattr(item, s)
-                       for s in columns]
+            if codelist_and_dsd:
+                # scheme is a (dimension or attribute, codelist) pair
+                dim_attr, scheme = scheme
+            # first row of a scheme, DSD-less codelist, conceptscheme etc.
+            if item is None:
+                # take the column attributes from the scheme itself
+                item = scheme
+            raw = [getattr(item, s) for s in columns]
             # Select language for international strings represented as dict
             translated = [s[lang] if lang in s
-                          else s.get('en') or ((s or None) and s.any()) for s in raw]
+                          else (s.get('en') or ((s or None) and s.any())) for s in raw]
             # for codelists, prepend dim_or_attr flag
-            if rows == 'codelist' and dsd:
-                if scheme in dim2cl.values():
+            if codelist_and_dsd:
+                if dim_attr in dim2cl:
                     translated.insert(0, 'D')
                 else:
                     translated.insert(0, 'A')
@@ -109,7 +104,7 @@ class Writer(BaseWriter):
                 return translated[0]
 
         def iter_keys(container):
-            if rows == 'codelist' and dsd:
+            if codelist_and_dsd:
                 if (constraint
                         and container[1] in dim2cl.values()):
                     result = (v for v in container[1].values()
@@ -121,28 +116,34 @@ class Writer(BaseWriter):
             return sorted(result, key=attrgetter('id'))
 
         def iter_schemes():
-            if rows == 'codelist' and dsd:
+            if codelist_and_dsd:
                 return chain(dim2cl.items(), attr2cl.items())
             else:
                 return content.values()
 
         def container2id(container, item):
-            if rows == 'codelist' and dsd:
+            if codelist_and_dsd:
                 # For first index level, get dimension or attribute ID instead of
                 # codelist ID
                 container_id = container[0].id
-                if container is item:
-                    item_id = container_id
-                else:
-                    item_id = item.id
+                # 2nd index col: first row
+                # contains the concept, all subsequent rows are codes.
+                item_id = item.id
             else:
+                # any other structure or codelist without DSD
                 container_id = container.id
-                if container is item:
-                    item_id = container.id
-                else:
-                    item_id = item.id
+                item_id = item.id if item else None  # None in first row
             return container_id, item_id
 
+        def row1_col2(container):
+            if codelist_and_dsd:
+                # return the concept of the dimension or attribute
+                # instead of the (dim, codelist) pair
+                return container[0].concept
+            # all other cases: return None as there is nothing
+            # interesting about, e.g. dataflow.
+            return None
+
         if rows == 'codelist':
             # Assuming a msg contains only one DSD
             try:
@@ -162,15 +163,19 @@ class Writer(BaseWriter):
                     constraint = constraint.constraints.any()
                 except AttributeError:
                     # So the Message must containe a constraint
-                    # the following is buggy: Should be linked to a dataflor,
+                    # the following is buggy: Should be linked to a dataflow,
                     # DSD or provision-agreement
                     constraint = source.constraints.any()
 
+        # pre-compute bool value to test for DSD-related codelists
+        codelist_and_dsd = (rows == 'codelist' and dsd)
+
         # allow `columns` arg to be a str
         if not isinstance(columns, (list, tuple)):
             columns = [columns]
-
-        content = getattr(source, rows)
+        # Get the structures to be written, e.g. codelist, dataflow,
+        # conceptscheme
+        content = getattr(source, rows)  # 'source' is the SDMX message
         # Distinguish hierarchical content consisting of a dict of dicts, and
         # flat content consisting of a dict of atomic model instances. In the former case,
         # the resulting DataFrame will have 2 index levels.
@@ -182,12 +187,12 @@ class Writer(BaseWriter):
             # we need the model instances as we want to gleen
             # from them other attributes for the dataframe columns.
             raw_tuples = chain.from_iterable(zip(
-                # 1st index level eg name of dimension
-                # represented by codelist
+                # 1st index level eg ID of dimension
+                # represented by codelist, or or ConceptScheme etc.
                 repeat(container),
                 # 2nd index level: first row in each codelist is the corresponding
                 # container id. The following rows are item ID's. .
-                chain((container,), iter_keys(container)))
+                chain((row1_col2(container),), iter_keys(container)))
                 for container in iter_schemes())
             # Now actually generate the index and related data for the columns
             raw_idx, data = zip(*[(container2id(i, j),
@@ -202,7 +207,7 @@ class Writer(BaseWriter):
             idx = PD.Index(raw_idx, name=rows)
         # For codelists, if there is a dsd, prepend 'dim_or_attr' as synthetic column
         # See corresponding insert in the make_columns function above
-        if rows == 'codelist' and dsd:
+        if codelist_and_dsd:
             # make local copy to avoid side effect
             columns = columns[:]
             columns.insert(0, 'dim_or_attr')
diff --git a/requirements.txt b/requirements.txt
index f9cf118..a51ae7e 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,8 +1,7 @@
-ipython>=4.0.0
-lxml>=3.3.0
 jsonpath-rw>=1.4.0
-    pytest>=2.8.0
+lxml>=3.3.0
 pandas>=0.18.0
+pytest>=2.8.0
 requests>=2.7
+setuptools>19.0
 Sphinx>=1.4.1
-setuptools > 19.0


Test patch:
diff --git a/tests/test_json.py b/tests/test_json.py
new file mode 100644
index 0000000..279819b
--- /dev/null
+++ b/tests/test_json.py
@@ -0,0 +1,125 @@
+from collections import OrderedDict
+import json
+import os
+from os.path import abspath, dirname, join
+from unittest import TestCase
+
+import pandas as pd
+from pandasdmx import Request
+
+
+# Construct the expected pandas representation of the sample data files
+sample_attrs = OrderedDict([
+    ('FREQ', 'D'),
+    ('CURRENCY_DENOM', 'EUR'),
+    ('EXR_TYPE', 'SP00'),
+    ('EXR_SUFFIX', 'A'),
+    ])
+
+
+def _idx(name, at_level=None, type=None):
+    """Return an index for dimension *name* in the sample data files.
+
+    If *at_level* is an integer, the contents of sample_attrs are added as
+    pd.MultiIndex level, with *name* in the position indicated by *at_level*.
+
+    If *name* is 'TIME_PERIOD' and *type* is pd.Period, a pd.PeriodIndex is
+    returned.
+    """
+    cls = pd.Index
+    kwargs = dict(name=name)
+
+    if name == 'CURRENCY':
+        data = ['NZD', 'RUB']
+    elif name == 'TIME_PERIOD':
+        data = ['2013-01-18', '2013-01-21']
+        if type == pd.Period:
+            cls = pd.PeriodIndex
+            kwargs['freq'] = 'D'
+
+    idx = cls(data, **kwargs)
+
+    if at_level is not None:
+        # Add sample_attrs as extra pd.MultiIndex levels on df.columns
+        names = list(sample_attrs.keys())
+        iterables = [[v] for v in sample_attrs.values()]
+        names.insert(at_level, name)
+        iterables.insert(at_level, idx)
+        return pd.MultiIndex.from_product(iterables, names=names)
+    else:
+        return idx
+
+
+sample_data = {
+    'action-delete': pd.DataFrame(
+        [[40.3426, 40.3000]],
+        index=_idx('CURRENCY')[1:],
+        columns=_idx('TIME_PERIOD', 1),
+        ),
+    'cross-section': pd.DataFrame(
+        [[1.5931, 1.5925], [40.3426, 40.3000]],
+        index=_idx('CURRENCY'),
+        columns=_idx('TIME_PERIOD', 1),
+        ),
+    'flat': pd.Series(
+        [1.5931, 1.5925, 40.3426, 40.3000],
+        index=pd.MultiIndex.from_product([[sample_attrs['FREQ']],
+                                          _idx('CURRENCY'),
+                                          [sample_attrs['CURRENCY_DENOM']],
+                                          [sample_attrs['EXR_TYPE']],
+                                          [sample_attrs['EXR_SUFFIX']],
+                                          _idx('TIME_PERIOD'),
+                                          ]),
+        ),
+    'time-series': pd.DataFrame(
+        [[1.5931, 40.3426], [1.5925, 40.3000]],
+        index=_idx('TIME_PERIOD', None, pd.Period),
+        columns=_idx('CURRENCY', 1),
+        ),
+    }
+
+
+class TestJSON(TestCase):
+
+    def setUp(self):
+        self.data_path = join(dirname(abspath(__file__)), 'data', 'json')
+
+    def _filepath(self, part):
+        return join(self.data_path, 'exr-%s.json' % part)
+
+    def test_samples_request(self):
+        """Test the samples from the SDMXJSON spec."""
+        req = Request()
+        for name, data in sample_data.items():
+            resp = req.get(fromfile=self._filepath(name))
+
+            df = resp.write()
+            assert df.equals(data), \
+                '\n'.join(map(str, [name, df.index, data.index,
+                                    getattr(df, 'columns', ''),
+                                    getattr(data, 'columns', '')]))
+
+    def test_read_as_str(self):
+        """Test the read_as_str() method."""
+        resp = Request().get(fromfile=self._filepath('flat'))
+        assert resp.header.id == '62b5f19d-f1c9-495d-8446-a3661ed24753'
+
+    def test_write_source(self):
+        """Test the write_source() method."""
+        req = Request()
+        for name in sample_data.keys():
+            orig_fn = self._filepath(name)
+            temp_fn = self._filepath(name + '-write-source')
+
+            # Read the message
+            resp = req.get(fromfile=orig_fn)
+
+            # Write to a temporary JSON file
+            resp.write_source(temp_fn)
+
+            # Read the two files and compare JSON (ignores ordering)
+            with open(orig_fn) as orig, open(temp_fn) as temp:
+                assert json.load(orig) == json.load(temp)
+
+            # Delete the temporary file
+            os.remove(temp_fn)