Repository: conda/conda

Issue:
Error when updating/installing packages: ValueError: No JSON object could be decoded
I am hitting this JSON-related `ValueError` basically when trying to update or install any package. For a MWE, I can run the following commands:

    $conda create -n testenv
    $source activate testenv
    $conda install python

The full error message I get is the following:

----

Solving environment: done

`$ /home/pablo/anaconda/bin/conda install python`

    Traceback (most recent call last):
      File "/home/pablo/anaconda/lib/python2.7/site-packages/conda/exceptions.py", line 722, in __call__
        return func(*args, **kwargs)
      File "/home/pablo/anaconda/lib/python2.7/site-packages/conda/cli/main.py", line 78, in _main
        exit_code = do_call(args, p)
      File "/home/pablo/anaconda/lib/python2.7/site-packages/conda/cli/conda_argparse.py", line 75, in do_call
        exit_code = getattr(module, func_name)(args, parser)
      File "/home/pablo/anaconda/lib/python2.7/site-packages/conda/cli/main_install.py", line 11, in execute
        install(args, parser, 'install')
      File "/home/pablo/anaconda/lib/python2.7/site-packages/conda/cli/install.py", line 239, in install
        handle_txn(progressive_fetch_extract, unlink_link_transaction, prefix, args, newenv)
      File "/home/pablo/anaconda/lib/python2.7/site-packages/conda/cli/install.py", line 256, in handle_txn
        unlink_link_transaction.display_actions(progressive_fetch_extract)
      File "/home/pablo/anaconda/lib/python2.7/site-packages/conda/core/link.py", line 706, in display_actions
        legacy_action_groups = self.make_legacy_action_groups(pfe)
      File "/home/pablo/anaconda/lib/python2.7/site-packages/conda/core/link.py", line 689, in make_legacy_action_groups
        pfe.prepare()
      File "/home/pablo/anaconda/lib/python2.7/site-packages/conda/common/io.py", line 402, in decorated
        return f(*args, **kwds)
      File "/home/pablo/anaconda/lib/python2.7/site-packages/conda/core/package_cache.py", line 508, in prepare
        for prec in self.link_precs)
      File "/home/pablo/anaconda/lib/python2.7/_abcoll.py", line 571, in update
        for key, value in other:
      File "/home/pablo/anaconda/lib/python2.7/site-packages/conda/core/package_cache.py", line 508, in <genexpr>
        for prec in self.link_precs)
      File "/home/pablo/anaconda/lib/python2.7/site-packages/conda/core/package_cache.py", line 402, in make_actions_for_record
        ), None)
      File "/home/pablo/anaconda/lib/python2.7/site-packages/conda/core/package_cache.py", line 399, in <genexpr>
        pcrec for pcrec in concat(PackageCache(pkgs_dir).query(pref_or_spec)
      File "/home/pablo/anaconda/lib/python2.7/site-packages/conda/core/package_cache.py", line 400, in <genexpr>
        for pkgs_dir in context.pkgs_dirs)
      File "/home/pablo/anaconda/lib/python2.7/site-packages/conda/core/package_cache.py", line 116, in query
        return (pcrec for pcrec in itervalues(self._package_cache_records) if pcrec == param)
      File "/home/pablo/anaconda/lib/python2.7/site-packages/conda/core/package_cache.py", line 209, in _package_cache_records
        self.load()
      File "/home/pablo/anaconda/lib/python2.7/site-packages/conda/core/package_cache.py", line 88, in load
        package_cache_record = self._make_single_record(base_name)
      File "/home/pablo/anaconda/lib/python2.7/site-packages/conda/core/package_cache.py", line 279, in _make_single_record
        index_json_record = read_index_json(extracted_package_dir)
      File "/home/pablo/anaconda/lib/python2.7/site-packages/conda/gateways/disk/read.py", line 112, in read_index_json
        record = IndexJsonRecord(**json.load(fi))
      File "/home/pablo/anaconda/lib/python2.7/json/__init__.py", line 291, in load
        **kw)
      File "/home/pablo/anaconda/lib/python2.7/json/__init__.py", line 339, in loads
        return _default_decoder.decode(s)
      File "/home/pablo/anaconda/lib/python2.7/json/decoder.py", line 364, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/home/pablo/anaconda/lib/python2.7/json/decoder.py", line 382, in raw_decode
        raise ValueError("No JSON object could be decoded")
    ValueError: No JSON object could be decoded

  environment variables:
                 CIO_TEST=<not set>
        CONDA_DEFAULT_ENV=testenv
             CONDA_PREFIX=/home/pablo/anaconda/envs/testenv
    CONDA_PROMPT_MODIFIER=(testenv)
         CONDA_PYTHON_EXE=/home/pablo/anaconda/bin/python
               CONDA_ROOT=/home/pablo/anaconda
              CONDA_SHLVL=1
            DEFAULTS_PATH=/usr/share/gconf/cinnamon.default.path
           MANDATORY_PATH=/usr/share/gconf/cinnamon.mandatory.path
                     PATH=/home/pablo/anaconda/envs/testenv/bin:/home/pablo/anaconda/bin:/home/p
                          ablo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:
                          /usr/games:/usr/local/games:/home/pablo/software_install/mangle/mangle
                          2.2/bin:/home/pablo/software_install/mangle/mangle2.2/scripts
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>
               WINDOWPATH=8

     active environment : testenv
    active env location : /home/pablo/anaconda/envs/testenv
            shell level : 1
       user config file : /home/pablo/.condarc
 populated config files : 
          conda version : 4.4.2
    conda-build version : 2.1.3
         python version : 2.7.13.final.0
       base environment : /home/pablo/anaconda  (writable)
           channel URLs : https://repo.continuum.io/pkgs/main/linux-64
                          https://repo.continuum.io/pkgs/main/noarch
                          https://repo.continuum.io/pkgs/free/linux-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/linux-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/linux-64
                          https://repo.continuum.io/pkgs/pro/noarch
          package cache : /home/pablo/anaconda/pkgs
                          /home/pablo/.conda/pkgs
       envs directories : /home/pablo/anaconda/envs
                          /home/pablo/.conda/envs
               platform : linux-64
             user-agent : conda/4.4.2 requests/2.14.2 CPython/2.7.13 Linux/3.16.0-38-generic ubuntu/14.04 glibc/2.19
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False


An unexpected error has occurred. Conda has prepared the above report.
If submitted, this report will be used by core maintainers to improve
future releases of conda.
Would you like conda to send this report to the core maintainers?
[y/N]: n

No report sent. To permanently opt-out, use

    $ conda config --set report_errors false

----

As I mentioned, I get the same error when trying to install other packages, or updating any existing ones (e.g. `$conda update anaconda` in my root environment).

I have tried cleaning everything with `$conda clean --all` (which actually listed lots of tarballs and packages that were cleaned), but I'm still getting the same error.

I have also run the code suggested by @kalefranz in #4419 and #4425 to identify problematic json files in the `conda-meta` folder, but got no output.

Thanks for your help on this!



Discussion:

Resolving patch:
diff --git a/conda/common/compat.py b/conda/common/compat.py
--- a/conda/common/compat.py
+++ b/conda/common/compat.py
@@ -47,9 +47,11 @@
 if PY3:  # pragma: py2 no cover
     from io import StringIO
     from itertools import zip_longest
+    from json import JSONDecodeError
 elif PY2:  # pragma: py3 no cover
     from cStringIO import StringIO
     from itertools import izip as zip, izip_longest as zip_longest
+    JSONDecodeError = ValueError
 
 StringIO = StringIO
 zip = zip
diff --git a/conda/core/package_cache.py b/conda/core/package_cache.py
--- a/conda/core/package_cache.py
+++ b/conda/core/package_cache.py
@@ -12,7 +12,8 @@
 from .._vendor.auxlib.collection import first
 from ..base.constants import CONDA_TARBALL_EXTENSION, PACKAGE_CACHE_MAGIC_FILE
 from ..base.context import context
-from ..common.compat import iteritems, itervalues, odict, text_type, with_metaclass
+from ..common.compat import (JSONDecodeError, iteritems, itervalues, odict, text_type,
+                             with_metaclass)
 from ..common.constants import NULL
 from ..common.io import ProgressBar, time_recorder
 from ..common.path import expand, url_to_path
@@ -34,7 +35,6 @@
 except ImportError:  # pragma: no cover
     from .._vendor.toolz.itertoolz import concat, concatv, groupby  # NOQA
 
-
 log = getLogger(__name__)
 
 
@@ -272,13 +272,25 @@ def _make_single_record(self, package_filename):
                 extracted_package_dir=extracted_package_dir,
             )
             return package_cache_record
-        except (IOError, OSError):
-            # no info/repodata_record.json exists
+        except (IOError, OSError, JSONDecodeError) as e:
+            # IOError / OSError if info/repodata_record.json doesn't exists
+            # JsonDecodeError if info/repodata_record.json is partially extracted or corrupted
+            #   python 2.7 raises ValueError instead of JsonDecodeError
+            #   ValueError("No JSON object could be decoded")
+            log.debug("unable to read %s\n  because %r",
+                      join(extracted_package_dir, 'info', 'repodata_record.json'), e)
+
             # try reading info/index.json
             try:
                 index_json_record = read_index_json(extracted_package_dir)
-            except (IOError, OSError):
-                # info/index.json doesn't exist either
+            except (IOError, OSError, JSONDecodeError) as e:
+                # IOError / OSError if info/index.json doesn't exist
+                # JsonDecodeError if info/index.json is partially extracted or corrupted
+                #   python 2.7 raises ValueError instead of JsonDecodeError
+                #   ValueError("No JSON object could be decoded")
+                log.debug("unable to read %s\n  because",
+                          join(extracted_package_dir, 'info', 'index.json'), e)
+
                 if isdir(extracted_package_dir) and not isfile(package_tarball_full_path):
                     # We have a directory that looks like a conda package, but without
                     # (1) info/repodata_record.json or info/index.json, and (2) a conda package
@@ -295,12 +307,17 @@ def _make_single_record(self, package_filename):
                         index_json_record = read_index_json(extracted_package_dir)
                     else:
                         index_json_record = read_index_json_from_tarball(package_tarball_full_path)
-                except (EOFError, ReadError):
+                except (EOFError, ReadError) as e:
                     # EOFError: Compressed file ended before the end-of-stream marker was reached
                     # tarfile.ReadError: file could not be opened successfully
+                    # We have a corrupted tarball. Remove the tarball so it doesn't affect
+                    # anything, and move on.
+                    log.debug("unable to extract info/index.json from %s\n  because %r",
+                              package_tarball_full_path, e)
                     rm_rf(package_tarball_full_path)
                     return None
 
+            # we were able to read info/index.json, so let's continue
             if isfile(package_tarball_full_path):
                 md5 = compute_md5sum(package_tarball_full_path)
             else:


Test patch: