Repository: pypa/pip

Issue:
Differentiating organic vs automated installations
**What's the problem this feature will solve?**
<!-- What are you trying to do, that you are unable to achieve with pip as it currently stands? -->

Currently, pip installation statistics are aggregated to the gCloud and made available on [libraries.io](https://libraries.io) and [pepy.tech](https://pepy.tech). A lot of effort has gone into these numbers, but thanks to automation, they mean less now than they did a few years ago. 

CI and other automation, combined with maybe a bit too much reliance on PyPI's central infrastructure, have inflated the download numbers and diluted the signal with noise.

**Describe the solution you'd like**
<!-- Clear and concise description of what you want to happen. -->

We could detect when pip is being used interactively (by checking if stdin is a tty or some other mechanism), and include that in the pip install request headers, to be included in the statistics generated by the server.

<!-- Provide examples of real world use cases that this would enable and how it solves the problem described above. -->

This would provide us with much cleaner data for highlighting actual community activity, instead of drowning in automation trends, overly favoring professionalized sectors of Python. Specifically, a library being manually installed 100 times may well indicate something much more interesting than a CI (or, unfortunately, a production) fleet installing a package 10,000 times.

**Additional context**
<!-- Add any other context, links, etc. about the feature here. -->

* I wasn't sure whether to file this on pip or on Warehouse, it seems kind of :chicken: / :egg: to me.
* I'm not really sure if/how other package indexes solve this, but would be very interested in hearing.
* As an arbitrary example, I happen to know Mozilla uses PyPI for quite a few relatively-internal packages. Granted, they're open-source and I'm happy to see some infrastructure synergy. But, picking at random, [mozlog](http://pepy.tech/project/mozlog) actually ranks ok for downloads, even though it's not a very broadly-useful package, and I'm pretty sure the data will show it's mostly Mozilla infrastructure downloading it.

Thanks for your attention and keep up the good work!


Discussion:
Note that because of https://github.com/pypa/linehaul/issues/30, the numbers on Google BigQuery may already be meaningless for answering some questions.
I agree. It would definitely be useful to have separation of automated vs direct usage.

Maybe pypa/packaging-problems would be a good place for it?
@mhsmith like Nathaniel pointed out, the lossage should be fairly uniform, so I think the numbers would still be somewhat representative, if we were collecting them on top of the leaky linehaul, that is :)

@pradyunsg Glad you agree! Given that I suspect (and suggested) a straightforward `pip` enhancement, I'd like to keep this issue open. That said, I may cross-post this there, if you think it would improve the visibility. Let me know if so!
I think the fundamental problem here is I don't think you can actually detect this reasonably. For instance, if someone manually runs a bash script (or even a tox command), we'd probably want that to be not set as automated-- but by default those things will not have a tty. On the flip side, you have things like Travis CI which I believe mimics a tty, so then Travis CI will look like like a manual install instead of automated.

On a theoretical level, I don't have any problem with the idea-- I just have never been able to think of a good way of actually differentiating the types of uses automatically.
If we want to detect running under CI, I think that's actually fairly easy, because CI systems tend to advertise that fact in the environment. Just checking for `"CI" in os.environ or "BUILD_ID" in os.environ or "BUILD_BUILDID" in os.environ` would probably catch 95% of cases (including at least [Travis-CI](https://docs.travis-ci.com/user/environment-variables/#Default-Environment-Variables), [Appveyro](https://www.appveyor.com/docs/environment-variables/), [Circle-CI](https://circleci.com/docs/2.0/env-vars/#built-in-environment-variables), [Jenkins](https://wiki.jenkins.io/display/JENKINS/Building+a+software+project#Buildingasoftwareproject-belowJenkinsSetEnvironmentVariables), [VSTS](https://docs.microsoft.com/en-us/vsts/pipelines/build/variables?view=vsts&tabs=batch#predefined-variables)).

Or if you want to get fancier, it looks like the ci-info package ([2.5 million weekly downloads](https://www.npmjs.com/package/ci-info)) has a fairly comprehensive list of envvars to check for: https://github.com/watson/ci-info/blob/master/index.js
(Looks like they're missing VSTS though.)
See https://github.com/The-Compiler/pytest-vw for a Python project that can detect CI.
Yea, it isn't difficult to detect whether you're running in a CI, on most CI services -- or for that matter even which one you're running on. We likely still won't know what %age of the non-CI runs are not automated but having a separation between CI/non-CI is a good start.

I don't know if we'd want to have any distinction between various CI services (logging NULL if we don't have the information, otherwise a string like "travis" representing the service).

I posted #6273 to start addressing this.
I'm going to leave this open for now as opposed to auto-closing for the purposes of discussing whether an additional key-value should be added to store the value of `isatty()`. The PR that was just merged stored the different info of whether something is known to be running in CI. 
FWIW, I pinged on #zuul on Freenode, to see if anyone there has inputs on how to detect running within Zuul. That said, better detection of that is not a blocker in any form.

I'd love to see a way for us to tell pip we're running in a CI. For context, Google has several custom CIs that wouldn't be detected by this code, so a flag or env var that's something like "PIP_IS_CI" would be really cool.
@theacodes if one can set an environment variable, wouldn't setting `CI=true` achieve just that? Or will that have an impact on other parts of the CI?
Yeah, it might have unintended consequences.

On Mon, Apr 29, 2019, 9:51 AM Mahmoud Hashemi <notifications@github.com>
wrote:

> @theacodes <https://github.com/theacodes> if one can set an environment
> variable, wouldn't setting CI=true achieve just that? Or will that have
> an impact on other parts of the CI?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pypa/pip/issues/5499#issuecomment-487657059>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AAB5I4565CGSWO5SZCHYFM3PS4RS7ANCNFSM4FEXCP4A>
> .
>

> I'd love to see a way for us to tell pip we're running in a CI. For context, Google has several custom CIs that wouldn't be detected by this code, so a flag or env var that's something like "PIP_IS_CI" would be really cool.

I think it'd be fine (and low maintenance) to support this. The implementation would just be a matter of adding `PIP_IS_CI` to the `CI_ENVIRONMENT_VARIABLES` variable here: https://github.com/pypa/pip/blob/5a00ac4130108605fb1bf51e0e771041664fa33b/src/pip/_internal/download.py#L80


> The implementation would just be a matter of adding `PIP_IS_CI` to the `CI_ENVIRONMENT_VARIABLES` variable

@theacodes If you could file a PR for this, that'd be great!
Is `PIP_IS_CI` recommended for non-CI automated installations?
For example, provisioning server via cloud-init or ansible.

Resolving patch:
diff --git a/src/pip/_internal/download.py b/src/pip/_internal/download.py
--- a/src/pip/_internal/download.py
+++ b/src/pip/_internal/download.py
@@ -95,6 +95,8 @@
     'BUILD_ID',
     # AppVeyor, CircleCI, Codeship, Gitlab CI, Shippable, Travis CI
     'CI',
+    # Explicit environment variable.
+    'PIP_IS_CI',
 )
 
 


Test patch:
diff --git a/tests/unit/test_download.py b/tests/unit/test_download.py
--- a/tests/unit/test_download.py
+++ b/tests/unit/test_download.py
@@ -73,6 +73,7 @@ def test_user_agent():
     ('BUILD_BUILDID', True),
     ('BUILD_ID', True),
     ('CI', True),
+    ('PIP_IS_CI', True),
     # Test a prefix substring of one of the variable names we use.
     ('BUILD', False),
 ])