Repository: piskvorky/gensim

Issue:
Set up Azure pipelines for gensim
@piskvorky basic config is ready https://github.com/menshikh-iv/gensim/pull/2, waiting when you activate `azure` for gensim (please make sure than pipeline runs in separate PR, after that I'll continue)

_Originally posted by @menshikh-iv in https://github.com/RaRe-Technologies/gensim/pull/2814#issuecomment-621950805_


Discussion:
I tried setting up Azure Pipelines and got as far as here:

![Screenshot (2)](https://user-images.githubusercontent.com/610412/80923406-e7f44900-8d83-11ea-9f76-91ed00e5fcd5.png)

Do I understand correctly the app wants read+write access to all my repos, including private ones? If so, that's a show stopper – at least for OAuth from my own `piskvorky` account.

I thought Azure Pipelines could be enabled for selected repos only.
I think I figured it out. Instead of logging in to Azure Pipelines website and then creating a Pipeline from there (= OAuth above = no go), I went to the Github marketplace and installed Azure Pipeline app from there. [Instructions](https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/github?view=azure-devops&tabs=yaml).

That allowed me to select which Github repositories I want. I selected all our open source repos.

@menshikh-iv Azure Pipelines is now active, I added both you and Misha as admins. I have no idea what you mean by `(please make sure than pipeline runs in separate PR, after that I'll continue)`.
@piskvorky azure pipelines aren't really active, see https://github.com/RaRe-Technologies/gensim/pull/2823 (no azure job spawned)

by `please make sure than pipeline runs in separate PR, after that I'll continue` I mean that you test it first (run any random build from gensim repository), so, please, fix that and make sure than my PR trigger azure build, after I'll continue.
If I understand correctly you want me to:

1. create a new github branch and add that azure `yml` file to it
2. create a new azure pipeline at https://dev.azure.com/rare-technologies/gensim-ci by selecting that github branch
3. ?do something about your PR?

I should get to it tonight. You and Misha are admins, so feel free to add the pipeline yourself if you like.
I want that you make my PR trigger azure pipeline (no need to create new one, just edit my PR)
I have no access to https://dev.azure.com/rare-technologies/gensim-ci, only you have an access.
UPD: something wired with microsoft auth system & permissions, let me dig in ...
That's weird. This is how it's set up:

![azureadmin](https://user-images.githubusercontent.com/610412/80947901-676c3180-8df1-11ea-9a14-718e7ef56a41.png)

I can open your link only If I don't log in (and have no settings controls)
![image](https://user-images.githubusercontent.com/15094669/80948260-65f33700-8dfa-11ea-8adb-aa7545f5652e.png)
Looks like you should add us to organization @piskvorky  (I have empty organization list)
I already added you to the org – otherwise I couldn't have added you to the project:

<img width="1440" alt="Screen Shot 2020-05-04 at 10 40 44" src="https://user-images.githubusercontent.com/610412/80949155-c0d56000-8df3-11ea-944f-d69b0df550a2.png">

Did you receive an email with the invite? Maybe in spam?
@piskvorky no emails (even in spam), still no access :(
Hm. I'll try to see if there's any help/support in Azure DevOps.
NLP <3

![image](https://user-images.githubusercontent.com/15094669/80950122-ef583880-8dfd-11ea-8023-ae997cbf5fff.png)

LOL you should know better than to expect much from "NLP chatbots".

Here: https://developercommunity.visualstudio.com/content/problem/1016416/users-added-but-have-no-access.html
@piskvorky I'm digging into the documentation and have an idea: can you try to fully delete us from azure and add though GitHub usernames (instead of emails)? 

Note:
- https://docs.microsoft.com/en-us/azure/devops/organizations/public/invite-users-public?view=azure-devops

OK. The Azure support team responded, I'll try that first.
> Note:

That's what I did before. I re-did it again now – did you receive an email from `AzureDevOps@microsoft.com` now?

@mpenkov how about you? Did you receive anything?
No, I changed subscription setting in profile to "get all emails", can you try to resend?
@menshikh-iv Invite resent:

<img width="1434" alt="Screen Shot 2020-05-04 at 14 09 59" src="https://user-images.githubusercontent.com/610412/80964344-fb012a80-8e10-11ea-9c24-6d99821d7552.png">

@piskvorky got them, I have an access
Good. Because adding Github usernames (instead of emails) is not possible. I tried and adding a new user requires their email.
@piskvorky still can't run build: looks like lack of permissions to RaRe organization, can you create default python pipeline thought https://dev.azure.com/rare-technologies/gensim-ci/_build yourself?
Unfortunately not; it's asking for those crazy OAuth permissions on `piskvorky` again.

What Azure permissions do you need? You should have the same as me (admin) = full access.
> What permissions do you need?

No idea, I'm getting the same OAuth (without options) as you
PAT tokens maybe https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/github?view=azure-devops&tabs=yaml ? This shouldn't be such crazy as oauth
I made you an admin for https://github.com/RaRe-Technologies/gensim, as well as admin for https://dev.azure.com/rare-technologies/gensim-ci. I don't see any higher permissions than that.
![image](https://user-images.githubusercontent.com/15094669/80966416-2722a880-8e1d-11ea-8c59-da07eae290e8.png)

No idea what's wrong here =/

![image](https://user-images.githubusercontent.com/15094669/80966857-0149d380-8e1e-11ea-907c-1664236a95e5.png)
moving forward (thanks to https://developercommunity.visualstudio.com/content/problem/642369/pipelines-creation-falls-back-to-github-oauth-auth.html thread), but that's not the end

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -264,7 +264,6 @@ def run(self):
     'testfixtures',
     'Morfessor==2.0.2a4',
     'python-Levenshtein >= 0.10.2',
-    'visdom >= 0.1.8, != 0.1.8.7',
     'scikit-learn',
     # The following packages are commented out because they don't install on Windows. So skip the
     # related tests in AppVeyor. We still test them in Linux via Travis, see linux_testenv below.
@@ -282,7 +281,8 @@ def run(self):
 #
 #   https://packaging.python.org/discussions/install-requires-vs-requirements/
 #
-docs_testenv = win_testenv + distributed_env + [
+visdom_req = ['visdom >= 0.1.8, != 0.1.8.7']
+docs_testenv = win_testenv + distributed_env + visdom_req + [
     'sphinx <= 2.4.4',  # avoid `sphinx >= 3.0` that breaks the build
     'sphinx-gallery',
     'sphinxcontrib.programoutput',
@@ -308,7 +308,7 @@ def run(self):
 
 # Add additional requirements for testing on Linux. We skip some tests on Windows,
 # because the libraries below are too tricky to install there.
-linux_testenv = win_testenv[:]
+linux_testenv = win_testenv[:] + visdom_req
 if sys.version_info >= (3, 7):
     # HACK: Installing tensorflow causes a segfault in Travis on py3.6. Other Pythons work – a mystery.
     # See https://github.com/RaRe-Technologies/gensim/pull/2814#issuecomment-621477948


Test patch: