Repository: pypa/pip

Issue:
Differentiating organic vs automated installations
**What's the problem this feature will solve?**
<!-- What are you trying to do, that you are unable to achieve with pip as it currently stands? -->

Currently, pip installation statistics are aggregated to the gCloud and made available on [libraries.io](https://libraries.io) and [pepy.tech](https://pepy.tech). A lot of effort has gone into these numbers, but thanks to automation, they mean less now than they did a few years ago. 

CI and other automation, combined with maybe a bit too much reliance on PyPI's central infrastructure, have inflated the download numbers and diluted the signal with noise.

**Describe the solution you'd like**
<!-- Clear and concise description of what you want to happen. -->

We could detect when pip is being used interactively (by checking if stdin is a tty or some other mechanism), and include that in the pip install request headers, to be included in the statistics generated by the server.

<!-- Provide examples of real world use cases that this would enable and how it solves the problem described above. -->

This would provide us with much cleaner data for highlighting actual community activity, instead of drowning in automation trends, overly favoring professionalized sectors of Python. Specifically, a library being manually installed 100 times may well indicate something much more interesting than a CI (or, unfortunately, a production) fleet installing a package 10,000 times.

**Additional context**
<!-- Add any other context, links, etc. about the feature here. -->

* I wasn't sure whether to file this on pip or on Warehouse, it seems kind of :chicken: / :egg: to me.
* I'm not really sure if/how other package indexes solve this, but would be very interested in hearing.
* As an arbitrary example, I happen to know Mozilla uses PyPI for quite a few relatively-internal packages. Granted, they're open-source and I'm happy to see some infrastructure synergy. But, picking at random, [mozlog](http://pepy.tech/project/mozlog) actually ranks ok for downloads, even though it's not a very broadly-useful package, and I'm pretty sure the data will show it's mostly Mozilla infrastructure downloading it.

Thanks for your attention and keep up the good work!


Discussion:
Note that because of https://github.com/pypa/linehaul/issues/30, the numbers on Google BigQuery may already be meaningless for answering some questions.
I agree. It would definitely be useful to have separation of automated vs direct usage.

Maybe pypa/packaging-problems would be a good place for it?
@mhsmith like Nathaniel pointed out, the lossage should be fairly uniform, so I think the numbers would still be somewhat representative, if we were collecting them on top of the leaky linehaul, that is :)

@pradyunsg Glad you agree! Given that I suspect (and suggested) a straightforward `pip` enhancement, I'd like to keep this issue open. That said, I may cross-post this there, if you think it would improve the visibility. Let me know if so!
I think the fundamental problem here is I don't think you can actually detect this reasonably. For instance, if someone manually runs a bash script (or even a tox command), we'd probably want that to be not set as automated-- but by default those things will not have a tty. On the flip side, you have things like Travis CI which I believe mimics a tty, so then Travis CI will look like like a manual install instead of automated.

On a theoretical level, I don't have any problem with the idea-- I just have never been able to think of a good way of actually differentiating the types of uses automatically.
If we want to detect running under CI, I think that's actually fairly easy, because CI systems tend to advertise that fact in the environment. Just checking for `"CI" in os.environ or "BUILD_ID" in os.environ or "BUILD_BUILDID" in os.environ` would probably catch 95% of cases (including at least [Travis-CI](https://docs.travis-ci.com/user/environment-variables/#Default-Environment-Variables), [Appveyro](https://www.appveyor.com/docs/environment-variables/), [Circle-CI](https://circleci.com/docs/2.0/env-vars/#built-in-environment-variables), [Jenkins](https://wiki.jenkins.io/display/JENKINS/Building+a+software+project#Buildingasoftwareproject-belowJenkinsSetEnvironmentVariables), [VSTS](https://docs.microsoft.com/en-us/vsts/pipelines/build/variables?view=vsts&tabs=batch#predefined-variables)).

Or if you want to get fancier, it looks like the ci-info package ([2.5 million weekly downloads](https://www.npmjs.com/package/ci-info)) has a fairly comprehensive list of envvars to check for: https://github.com/watson/ci-info/blob/master/index.js
(Looks like they're missing VSTS though.)
See https://github.com/The-Compiler/pytest-vw for a Python project that can detect CI.
Yea, it isn't difficult to detect whether you're running in a CI, on most CI services -- or for that matter even which one you're running on. We likely still won't know what %age of the non-CI runs are not automated but having a separation between CI/non-CI is a good start.

I don't know if we'd want to have any distinction between various CI services (logging NULL if we don't have the information, otherwise a string like "travis" representing the service).


Resolving patch:
diff --git a/src/pip/_internal/download.py b/src/pip/_internal/download.py
--- a/src/pip/_internal/download.py
+++ b/src/pip/_internal/download.py
@@ -72,6 +72,34 @@
 logger = logging.getLogger(__name__)
 
 
+# These are environment variables present when running under various
+# CI systems.  For each variable, some CI systems that use the variable
+# are indicated.  The collection was chosen so that for each of a number
+# of popular systems, at least one of the environment variables is used.
+# This list is used to provide some indication of and lower bound for
+# CI traffic to PyPI.  Thus, it is okay if the list is not comprehensive.
+# For more background, see: https://github.com/pypa/pip/issues/5499
+CI_ENVIRONMENT_VARIABLES = (
+    # Azure Pipelines
+    'BUILD_BUILDID',
+    # Jenkins
+    'BUILD_ID',
+    # AppVeyor, CircleCI, Codeship, Gitlab CI, Shippable, Travis CI
+    'CI',
+)
+
+
+def looks_like_ci():
+    # type: () -> bool
+    """
+    Return whether it looks like pip is running under CI.
+    """
+    # We don't use the method of checking for a tty (e.g. using isatty())
+    # because some CI systems mimic a tty (e.g. Travis CI).  Thus that
+    # method doesn't provide definitive information in either direction.
+    return any(name in os.environ for name in CI_ENVIRONMENT_VARIABLES)
+
+
 def user_agent():
     """
     Return a string representing the user agent.
@@ -135,6 +163,12 @@ def user_agent():
     if setuptools_version is not None:
         data["setuptools_version"] = setuptools_version
 
+    # Use None rather than False so as not to give the impression that
+    # pip knows it is not being run under CI.  Rather, it is a null or
+    # inconclusive result.  Also, we include some value rather than no
+    # value to make it easier to know that the check has been run.
+    data["ci"] = True if looks_like_ci() else None
+
     return "{data[installer][name]}/{data[installer][version]} {json}".format(
         data=data,
         json=json.dumps(data, separators=(",", ":"), sort_keys=True),


Test patch:
diff --git a/tests/unit/test_download.py b/tests/unit/test_download.py
--- a/tests/unit/test_download.py
+++ b/tests/unit/test_download.py
@@ -10,8 +10,8 @@
 
 import pip
 from pip._internal.download import (
-    PipSession, SafeFileCache, path_to_url, unpack_file_url, unpack_http_url,
-    url_to_path,
+    CI_ENVIRONMENT_VARIABLES, PipSession, SafeFileCache, path_to_url,
+    unpack_file_url, unpack_http_url, url_to_path,
 )
 from pip._internal.exceptions import HashMismatch
 from pip._internal.models.link import Link
@@ -50,8 +50,41 @@ def _fake_session_get(*args, **kwargs):
         rmtree(temp_dir)
 
 
+def get_user_agent():
+    return PipSession().headers["User-Agent"]
+
+
 def test_user_agent():
-    PipSession().headers["User-Agent"].startswith("pip/%s" % pip.__version__)
+    user_agent = get_user_agent()
+
+    assert user_agent.startswith("pip/%s" % pip.__version__)
+
+
+@pytest.mark.parametrize('name, expected_like_ci', [
+    ('BUILD_BUILDID', True),
+    ('BUILD_ID', True),
+    ('CI', True),
+    # Test a prefix substring of one of the variable names we use.
+    ('BUILD', False),
+])
+def test_user_agent__ci(monkeypatch, name, expected_like_ci):
+    # Delete the variable names we use to check for CI to prevent the
+    # detection from always returning True in case the tests are being run
+    # under actual CI.  It is okay to depend on CI_ENVIRONMENT_VARIABLES
+    # here (part of the code under test) because this setup step can only
+    # prevent false test failures.  It can't cause a false test passage.
+    for ci_name in CI_ENVIRONMENT_VARIABLES:
+        monkeypatch.delenv(ci_name, raising=False)
+
+    # Confirm the baseline before setting the environment variable.
+    user_agent = get_user_agent()
+    assert '"ci":null' in user_agent
+    assert '"ci":true' not in user_agent
+
+    monkeypatch.setenv(name, 'true')
+    user_agent = get_user_agent()
+    assert ('"ci":true' in user_agent) == expected_like_ci
+    assert ('"ci":null' in user_agent) == (not expected_like_ci)
 
 
 class FakeStream(object):