Repository: oppia/oppia

Issue:
oppia_tools Invalid Syntax
when running python -m scripts.start ' I got this error

```
File "/Users/nikil/Desktop/opensource/oppia/scripts/start.py", line 31, in <module>
    from . import install_third_party_libs
  File "scripts/install_third_party_libs.py", line 42, in <module>
    import python_utils  # isort:skip
  File "python_utils.py", line 34, in <module>
    import yaml  # isort:skip
  File "/Users/nikil/Desktop/opensource/oppia/../oppia_tools/pyyaml-5.1.2/yaml/__init__.py", line 374
    class YAMLObject(metaclass=YAMLObjectMetaclass):
                              ^
SyntaxError: invalid syntax
```

How can solve this issue?

oppia_tools Invalid Syntax
when running python -m scripts.start ' I got this error

```
File "/Users/nikil/Desktop/opensource/oppia/scripts/start.py", line 31, in <module>
    from . import install_third_party_libs
  File "scripts/install_third_party_libs.py", line 42, in <module>
    import python_utils  # isort:skip
  File "python_utils.py", line 34, in <module>
    import yaml  # isort:skip
  File "/Users/nikil/Desktop/opensource/oppia/../oppia_tools/pyyaml-5.1.2/yaml/__init__.py", line 374
    class YAMLObject(metaclass=YAMLObjectMetaclass):
                              ^
SyntaxError: invalid syntax
```

How can solve this issue?



Discussion:
Hey @Nikil99 Seems like a pip version issue...
hey @Showtim3  How do i resolve it ??
@kevintab95 I would like to work on this issue.

This error is popping up because your system's default pip is pip3. As a temporary fix, you can change the default pip and get yourself going. You can find how to do this [here](https://stackoverflow.com/questions/42871090/change-default-ubuntu-pip-to-pip2-7/51275052).

For the permanent fix, changing `pip` to `pip2` in [install_third_party_libs.py](https://github.com/oppia/oppia/blob/develop/scripts/install_third_party_libs.py) will make things work out even if the default pip for any system is pip3.

Hi @ShivanshRakesh, sounds good. I've assigned this issue to you. :)
I'm not sure if this is an appropriate change. Wouldn't this cause issues for people who do not have their pip installation aliased as `pip2`? I believe the better solution is to encourage `virtualenv` environments, which creates a correct linkage of `pip` to version that was used to create the `virtualenv`.
I agree in that simply renaming isn't an appropriate change. We're in a process where we want our scripts to eventually adapt Python 3 as well, so that's an additional reason for not simply renaming. 

@Nikil99 can you provide more details about your machine? (ie. OS and version)
@kevinlee12 Okay.. got your point. So, instead of replacing `pip` with `pip2`, can we check for which pip is being used, by `pip.__file__`? If it is `pip2`, it will contain `python2`, else it won't. If it does, then it is fine, else we can try checking for `pip2` explicitly. 
Later, if we wish to switch to python3 also, the condition can easily be modified.

Also, even I was facing this issue, which got resolved by changing pip to pip2. I am running **Ubuntu 18.04.3 LTS**.
Hmm, that's interesting. I'm running on 19.10 and don't have those issues, but good to know. 

As for the solution, that sounds like a better plan. As for checking `pip.__file__`, what does that entail?
We just need to import pip, which we're already doing. `pip.__file__` returns the location of pip which was imported. For ex, it returns the following in my case:

> \>\>\> import pip
 \>\>\> pip.\_\_file\_\_
 '/usr/local/lib/python2.7/dist-packages/pip/\_\_init\_\_.pyc'

So we can check which pip is being used from the python version.
okay, let's go with that approach.
I just tried the way you proposed and found there is actually no module named `pip2`. I tried `import pip2` on Debian and Windows and all I got is `ImportError: No module named pip2`.

```shell
# Debian in virtual env
(venv) ➜  oppia git:(convert-e2e-to-python*)python
Python 2.7.15+ (default, Oct  7 2019, 17:39:04)
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pip2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named pip2

# Debian in default env
➜  oppia git:(convert-e2e-to-python*)python
Python 2.7.15+ (default, Oct  7 2019, 17:39:04)
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pip2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named pip2

# Windows 
C:\> py -2
Python 2.7.17 (v2.7.17:c2f86d86e6, Oct 19 2019, 21:01:17) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pip2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named pip2
```

May I know why we can not just use `virtualenv`? My personal feeling is some responsibilities should be on the scripts while some should be on the developer. 
@cuichenli One thing that you pointed out right is that we should not be importing `pip2`, since it is not a module. Will surely correct that.
**We are not checking for some python module 'pip2', instead, we are checking for the command line pip2.** That is why we are first checking for version of `pip` and if the default pip is not `pip2`, we are checking for it explicitly, in order to ensure that if `pip2` is installed on the system, we are using it. 

As far as the use of `virtualenv` is concerned, yes, we can definitely use that. But then again, if we can do something with the code to make the installation easier, we should, right? One of the reasons I am taking this up is that even I was stuck on this issue for a couple of days, which finally got resolved by replacing `pip` with `pip2` in `install_third_party_libs.py` wherever there were command line calls. But then, I made a mistake by importing `pip2`. Thanks for pointing that out!
Hey @Nikil99 Seems like a pip version issue...
hey @Showtim3  How do i resolve it ??
@kevintab95 I would like to work on this issue.

This error is popping up because your system's default pip is pip3. As a temporary fix, you can change the default pip and get yourself going. You can find how to do this [here](https://stackoverflow.com/questions/42871090/change-default-ubuntu-pip-to-pip2-7/51275052).

For the permanent fix, changing `pip` to `pip2` in [install_third_party_libs.py](https://github.com/oppia/oppia/blob/develop/scripts/install_third_party_libs.py) will make things work out even if the default pip for any system is pip3.

Hi @ShivanshRakesh, sounds good. I've assigned this issue to you. :)
I'm not sure if this is an appropriate change. Wouldn't this cause issues for people who do not have their pip installation aliased as `pip2`? I believe the better solution is to encourage `virtualenv` environments, which creates a correct linkage of `pip` to version that was used to create the `virtualenv`.
I agree in that simply renaming isn't an appropriate change. We're in a process where we want our scripts to eventually adapt Python 3 as well, so that's an additional reason for not simply renaming. 

@Nikil99 can you provide more details about your machine? (ie. OS and version)
@kevinlee12 Okay.. got your point. So, instead of replacing `pip` with `pip2`, can we check for which pip is being used, by `pip.__file__`? If it is `pip2`, it will contain `python2`, else it won't. If it does, then it is fine, else we can try checking for `pip2` explicitly. 
Later, if we wish to switch to python3 also, the condition can easily be modified.

Also, even I was facing this issue, which got resolved by changing pip to pip2. I am running **Ubuntu 18.04.3 LTS**.
Hmm, that's interesting. I'm running on 19.10 and don't have those issues, but good to know. 

As for the solution, that sounds like a better plan. As for checking `pip.__file__`, what does that entail?
We just need to import pip, which we're already doing. `pip.__file__` returns the location of pip which was imported. For ex, it returns the following in my case:

> \>\>\> import pip
 \>\>\> pip.\_\_file\_\_
 '/usr/local/lib/python2.7/dist-packages/pip/\_\_init\_\_.pyc'

So we can check which pip is being used from the python version.
okay, let's go with that approach.
I just tried the way you proposed and found there is actually no module named `pip2`. I tried `import pip2` on Debian and Windows and all I got is `ImportError: No module named pip2`.

```shell
# Debian in virtual env
(venv) ➜  oppia git:(convert-e2e-to-python*)python
Python 2.7.15+ (default, Oct  7 2019, 17:39:04)
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pip2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named pip2

# Debian in default env
➜  oppia git:(convert-e2e-to-python*)python
Python 2.7.15+ (default, Oct  7 2019, 17:39:04)
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pip2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named pip2

# Windows 
C:\> py -2
Python 2.7.17 (v2.7.17:c2f86d86e6, Oct 19 2019, 21:01:17) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pip2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named pip2
```

May I know why we can not just use `virtualenv`? My personal feeling is some responsibilities should be on the scripts while some should be on the developer. 
@cuichenli One thing that you pointed out right is that we should not be importing `pip2`, since it is not a module. Will surely correct that.
**We are not checking for some python module 'pip2', instead, we are checking for the command line pip2.** That is why we are first checking for version of `pip` and if the default pip is not `pip2`, we are checking for it explicitly, in order to ensure that if `pip2` is installed on the system, we are using it. 

As far as the use of `virtualenv` is concerned, yes, we can definitely use that. But then again, if we can do something with the code to make the installation easier, we should, right? One of the reasons I am taking this up is that even I was stuck on this issue for a couple of days, which finally got resolved by replacing `pip` with `pip2` in `install_third_party_libs.py` wherever there were command line calls. But then, I made a mistake by importing `pip2`. Thanks for pointing that out!

Resolving patch:
diff --git a/scripts/install_third_party_libs.py b/scripts/install_third_party_libs.py
--- a/scripts/install_third_party_libs.py
+++ b/scripts/install_third_party_libs.py
@@ -22,19 +22,20 @@
 import os
 import shutil
 import subprocess
+import sys
 
 # These libraries need to be installed before running or importing any script.
 TOOLS_DIR = os.path.join('..', 'oppia_tools')
 # Download and install pyyaml.
 if not os.path.exists(os.path.join(TOOLS_DIR, 'pyyaml-5.1.2')):
     subprocess.check_call([
-        'pip', 'install', 'pyyaml==5.1.2', '--target',
+        sys.executable, '-m', 'pip', 'install', 'pyyaml==5.1.2', '--target',
         os.path.join(TOOLS_DIR, 'pyyaml-5.1.2')])
 
 # Download and install future.
 if not os.path.exists(os.path.join('third_party', 'future-0.17.1')):
     subprocess.check_call([
-        'pip', 'install', 'future==0.17.1', '--target',
+        sys.executable, '-m', 'pip', 'install', 'future==0.17.1', '--target',
         os.path.join('third_party', 'future-0.17.1')])
 
 # pylint: disable=wrong-import-position
@@ -123,8 +124,8 @@ def pip_install(package, version, install_path):
         raise Exception
 
     subprocess.check_call([
-        'pip', 'install', '%s==%s' % (package, version), '--target',
-        install_path])
+        sys.executable, '-m', 'pip', 'install', '%s==%s' % (package, version),
+        '--target', install_path])
 
 
 def install_skulpt(parsed_args):


Test patch:
diff --git a/scripts/script_import_test.py b/scripts/script_import_test.py
--- a/scripts/script_import_test.py
+++ b/scripts/script_import_test.py
@@ -27,6 +27,7 @@
 
 import os
 import subprocess
+import sys
 
 from core.tests import test_utils
 
@@ -64,6 +65,7 @@ def mock_exists(unused_path):
             from scripts import install_third_party_libs # pylint: disable=unused-variable
         self.assertEqual(
             self.commands, [
-                'pip', 'install', 'pyyaml==5.1.2', '--target',
-                '../oppia_tools/pyyaml-5.1.2', 'pip', 'install',
-                'future==0.17.1', '--target', 'third_party/future-0.17.1'])
+                sys.executable, '-m', 'pip', 'install', 'pyyaml==5.1.2',
+                '--target', '../oppia_tools/pyyaml-5.1.2', sys.executable,
+                '-m', 'pip', 'install', 'future==0.17.1', '--target',
+                'third_party/future-0.17.1'])