Repository: oppia/oppia

Issue:
[Setup issue - macOS with M1 chipset] Running "python -m scripts.start" results in "Error installing package" error
I have successfully installed oppia on an Ubuntu 20.04 OS instance on an HP machine (however this machine is not overly stable) 

I am now trying to install oppia on my new macbook air 2020 with the M1 chipset with macOS Big Sur v11.1.

**Python version:** 2.7.16

I have followed all instructions including deleting specific folders, tried installing version 20.3.3 of pip, then re-running "python -m scripts.start" as well as all relevant troubleshooting instructions, however still receive the following error:

**Error received:**
Checking if enum34 is installed.
Checking if pip is installed on the local machine
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
ERROR: Could not install packages due to an EnvironmentError: [('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/enum/LICENSE', '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/LICENSE', "[Errno 1] Operation not permitted: '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/LICENSE'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/enum/__init__.py', '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/__init__.py', "[Errno 1] Operation not permitted: '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/enum/test.py', '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/test.py', "[Errno 1] Operation not permitted: '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/test.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/enum/README', '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/README', "[Errno 1] Operation not permitted: '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/README'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/enum/__init__.pyc', '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/__init__.pyc', "[Errno 1] Operation not permitted: '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/enum/doc/enum.pdf', '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/doc/enum.pdf', "[Errno 1] Operation not permitted: '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/doc/enum.pdf'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/enum/doc/enum.rst', '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/doc/enum.rst', "[Errno 1] Operation not permitted: '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/doc/enum.rst'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/enum/doc', '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/doc', "[Errno 1] Operation not permitted: '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/doc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/enum/test.pyc', '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/test.pyc', "[Errno 1] Operation not permitted: '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/test.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/enum', '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n', "[Errno 1] Operation not permitted: '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n'")]

WARNING: You are using pip version 19.3.1; however, version 20.3.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Refer to https://github.com/oppia/oppia/wiki/Troubleshooting
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/shellymutu-grigg/opensource/oppia/scripts/start.py", line 33, in <module>
    install_third_party_libs.main()
  File "scripts/install_third_party_libs.py", line 236, in main
    ensure_system_python_libraries_are_installed(package, version)
  File "scripts/install_third_party_libs.py", line 201, in ensure_system_python_libraries_are_installed
    install_backend_python_libs.pip_install_to_system(package, version)
  File "scripts/install_backend_python_libs.py", line 416, in pip_install_to_system
    'install', '%s==%s' % (package, version)])
  File "scripts/install_backend_python_libs.py", line 393, in _run_pip_command
    raise Exception('Error installing package')
Exception: Error installing package

If someone could assist, that would be very much appreciated.
[Setup issue - macOS with M1 chipset] Running "python -m scripts.start" results in "Error installing package" error
I have successfully installed oppia on an Ubuntu 20.04 OS instance on an HP machine (however this machine is not overly stable) 

I am now trying to install oppia on my new macbook air 2020 with the M1 chipset with macOS Big Sur v11.1.

**Python version:** 2.7.16

I have followed all instructions including deleting specific folders, tried installing version 20.3.3 of pip, then re-running "python -m scripts.start" as well as all relevant troubleshooting instructions, however still receive the following error:

**Error received:**
Checking if enum34 is installed.
Checking if pip is installed on the local machine
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
ERROR: Could not install packages due to an EnvironmentError: [('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/enum/LICENSE', '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/LICENSE', "[Errno 1] Operation not permitted: '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/LICENSE'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/enum/__init__.py', '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/__init__.py', "[Errno 1] Operation not permitted: '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/enum/test.py', '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/test.py', "[Errno 1] Operation not permitted: '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/test.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/enum/README', '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/README', "[Errno 1] Operation not permitted: '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/README'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/enum/__init__.pyc', '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/__init__.pyc', "[Errno 1] Operation not permitted: '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/enum/doc/enum.pdf', '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/doc/enum.pdf', "[Errno 1] Operation not permitted: '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/doc/enum.pdf'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/enum/doc/enum.rst', '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/doc/enum.rst', "[Errno 1] Operation not permitted: '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/doc/enum.rst'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/enum/doc', '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/doc', "[Errno 1] Operation not permitted: '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/doc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/enum/test.pyc', '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/test.pyc', "[Errno 1] Operation not permitted: '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n/test.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/enum', '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n', "[Errno 1] Operation not permitted: '/private/var/folders/w2/knd48p2d1nn_wyjdqwkx00sc0000gn/T/pip-uninstall-5lhb0n'")]

WARNING: You are using pip version 19.3.1; however, version 20.3.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Refer to https://github.com/oppia/oppia/wiki/Troubleshooting
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/shellymutu-grigg/opensource/oppia/scripts/start.py", line 33, in <module>
    install_third_party_libs.main()
  File "scripts/install_third_party_libs.py", line 236, in main
    ensure_system_python_libraries_are_installed(package, version)
  File "scripts/install_third_party_libs.py", line 201, in ensure_system_python_libraries_are_installed
    install_backend_python_libs.pip_install_to_system(package, version)
  File "scripts/install_backend_python_libs.py", line 416, in pip_install_to_system
    'install', '%s==%s' % (package, version)])
  File "scripts/install_backend_python_libs.py", line 393, in _run_pip_command
    raise Exception('Error installing package')
Exception: Error installing package

If someone could assist, that would be very much appreciated.


Discussion:
@shellymutu-grigg Thanks for creating this issue! Sorry we could not take a look at this earlier. I have assigned it to dev-workflow team member now, and hopefully we would find a fix for this soon. Also great job on providing us all the details.
@shellymutu-grigg, did you find any fix for this?
Although I tried setting up the oppia env using Rosetta 2, I came across a different error:

```
Failed to start server on port 6379, exiting ...
INFORMATION

Cleaning up the servers.

Cleaning up the redis_servers.
Could not connect to Redis at 127.0.0.1:6379: Connection refused
```

 (If you find any fix related to these, please let me know) 

@krishita30j @shellymutu-grigg  can u check if you can install a python package using the `pip` cmd (any package that Oppia uses). I just want to rule out the possibility that it's an architecture issue.
@gp201, yeah I can install python packages seperately using pip.
> @shellymutu-grigg, did you find any fix for this?
> Although I tried setting up the oppia env using Rosetta 2, I came across a different error:
> 
> ```
> Failed to start server on port 6379, exiting ...
> INFORMATION
> 
> Cleaning up the servers.
> 
> Cleaning up the redis_servers.
> Could not connect to Redis at 127.0.0.1:6379: Connection refused
> ```
> 
> (If you find any fix related to these, please let me know)

@krishita30j have you tried [this](https://stackoverflow.com/questions/42857551/could-not-connect-to-redis-at-127-0-0-16379-connection-refused-with-homebrew).
@DubeySandeep @shellymutu-grigg
## Findings
Given below are the findings based on the offline discussion with @krishita30j :
All findings given below were run on [Rosetta 2](https://support.apple.com/en-us/HT211861)
- The Redis package is installed using pip in the cmd line, but it does not run (was installed by oppia when the `python -m scripts.start` cmd is run).
- The Redis server is installed successfully using `pip3` (meant for python 3) cmd and also runs.
- The reason the packages don't get downloaded is because of the architecture. Not all packages are able to run on ARM architecture (M1 chips).
## Current Solution:
1. Install Rosetta 2 try [this](https://www.google.com/url?q=https://stackoverflow.com/questions/64882584/how-to-run-the-homebrew-installer-under-rosetta-2-on-m1-macbook%23:~:text%3DYou%2520can%2520run%2520Terminal%2520with,%2522Open%2520using%2520Rosetta%2522%2520option&sa=D&source=hangouts&ust=1611504651223000&usg=AFQjCNFhJysGCAFdzLswFuK5JVoxjs6CwQ)
2. Inside Rosetta perform all the installations/prerequisites.
Note: If `sudo easy_install pyyaml` does not work try using `pip3 install pyyaml`.
3. Open the rosetta terminal and download redis server using `pip3 install redis`.
4. Run redis-server with the following cmd: `redis-server` (**Don't stop the redis server**)
5. Open another rosetta terminal and run `python -m scripts.start`

EDIT: Added step-2
@sajalasati @gp201, Thanks a lot for looking into this issue, can you please confirm whether the issue is resolved now? (If yes, then I'll go ahead and add the fix to the troubleshooting wiki page and close this issue.)
@DubeySandeep The steps above were used to resolve the issue on @krishita30j computer.
@krishita30j can you cross-check the details?
> @DubeySandeep @shellymutu-grigg
> 
> ## Findings
> Given below are the findings based on the offline discussion with @krishita30j :
> All findings given below were run on [Rosetta 2](https://support.apple.com/en-us/HT211861)
> 
> * The Redis package is installed using pip in the cmd line, but it does not run (was installed by oppia when the `python -m scripts.start` cmd is run).
> * The Redis server is installed successfully using `pip3` (meant for python 3) cmd and also runs.
> * The reason the packages don't get downloaded is because of the architecture. Not all packages are able to run on ARM architecture (M1 chips).
> 
> ## Current Solution:
> 1. Install Rosetta 2 try [this](https://www.google.com/url?q=https://stackoverflow.com/questions/64882584/how-to-run-the-homebrew-installer-under-rosetta-2-on-m1-macbook%23:~:text%3DYou%2520can%2520run%2520Terminal%2520with,%2522Open%2520using%2520Rosetta%2522%2520option&sa=D&source=hangouts&ust=1611504651223000&usg=AFQjCNFhJysGCAFdzLswFuK5JVoxjs6CwQ)
> 2. Inside Rosetta perform all the installations/prerequisites.
>    Note: If `sudo easy_install pyyaml` does not work try using `brew`.
> 3. Open the rosetta terminal and download redis server using `pip3 install redis`.
> 4. Run redis-server with the following cmd: `redis-server` (**Don't stop the redis server**)
> 5. Open another rosetta terminal and run `python -m scripts.start`
> 
> EDIT: Added step-2

@gp201, I checked it again and it seems like `brew` won't work to install `pyyaml`. Instead, use `pip3 install pyyaml`. Other than this, everything else looks good.
@DubeySandeep looks like the fix is correct now. It can be added to the troubleshooting page now.
@gp201 @DubeySandeep @sajalasati while running the e2e tests locally, this error showed up (while running e2e test, I started the redis-server in another tab):

<img width="1440" alt="Screenshot 2021-01-30 at 7 27 55 PM" src="https://user-images.githubusercontent.com/66058206/106361470-0bb0e380-6344-11eb-8a91-ad2e9a564724.png">

<img width="1440" alt="Screenshot 2021-01-30 at 7 28 00 PM" src="https://user-images.githubusercontent.com/66058206/106361474-0f446a80-6344-11eb-8c80-48a28cf8f670.png">

And while running backend test, this error showed up:

<img width="1440" alt="Screenshot 2021-01-30 at 9 32 21 PM" src="https://user-images.githubusercontent.com/66058206/106361480-1ec3b380-6344-11eb-9f08-5c837ef023d1.png">

Can you please assist me in fixing this?
@U8NWXD PTAL at this issue. It is wrt to e2e test and backend test running locally in Mac with M1 chips
Hi,

I have tried these steps but receive the following error at step 2:

ERROR: Could not find a version that satisfies the requirement redis-server
ERROR: No matching distribution found for redis-server



> On 24/01/2021, at 05:10, PRANEETH <notifications@github.com> wrote:
> 
> 
> Findings
> 
> Given below are the findings based on the offline discussion with @krishita30j <https://github.com/krishita30j> :
> All findings given below were run on Rosetta 2 <https://support.apple.com/en-us/HT211861>
> The Redis package is installed using pip in the cmd line, but it does not run (was installed by oppia when the python -m scripts.start cmd is run).
> The Redis server is installed successfully using pip3 (meant for python 3) cmd and also runs.
> Current Solution:
> 
> Install Rosetta 2
> Open the rosetta terminal and download redis server using pip3 install redis-server.
> Run redis-server with the following cmd: redis-server (Don't stop the redis server)
> Open another rosetta terminal and run python -m scripts.start
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub <https://github.com/oppia/oppia/issues/11584#issuecomment-766111198>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANEUMZMBNM5KKS3DERFWX2DS3LYIZANCNFSM4VRVWSCQ>.
> 


@shellymutu-grigg can you try just executing the `scripts.run_e2e_tests_test.RunE2ETestsTests.test_work_with_non_ascii_chars` backend test? I suspect that's the one that's failing. You can run this test like so:

```console
$ python -m scripts.run_backend_tests --test_target scripts.run_e2e_tests_test.RunE2ETestsTests.test_work_with_non_ascii_chars
```
@U8NWXD sorry I should have been more clear, @shellymutu-grigg is facing a setup issue and @krishita30j is facing the e2e test and backend issue
@U8NWXD the `scripts.run_e2e_tests_test.RunE2ETestsTests.test_work_with_non_ascii_chars` test passes successfully. 
<img width="757" alt="Screenshot 2021-01-31 at 12 17 21 PM" src="https://user-images.githubusercontent.com/66058206/106377447-0fd01600-63c3-11eb-8d83-2021a82a64e0.png">


@krishita30j interesting. What about the whole `run_e2e_tests_test` module?
@U8NWXD Basic Info:

1. During setup the redis server was not running. The [current fix](https://github.com/oppia/oppia/issues/11584#issuecomment-766111198) is given above.

2. @krishita30j has already tried to run the `redis-server` in a different (rosetta 2) terminal and execute the tests in a separate (rosetta 2) terminal.
@U8NWXD running the whole `run_e2e_tests_test` module results in an ImportError.

<img width="1440" alt="Screenshot 2021-02-01 at 12 10 10 PM" src="https://user-images.githubusercontent.com/66058206/106424087-9cdca300-6487-11eb-8e36-3b5961a0bed9.png">

@krishita30j Sorry, I didn't mean running the module directly. I meant running all the backend tests in that module like this:

```console
python -m scripts.run_backend_tests --test_target scripts.run_e2e_tests_test
```
@U8NWXD, the tests pass successfully:
<img width="631" alt="Screenshot 2021-02-03 at 11 15 16 AM" src="https://user-images.githubusercontent.com/66058206/106704047-69c61b00-6611-11eb-94f3-34d176e38262.png">

Thanks @krishita30j! I looked into the code causing this error:

https://github.com/oppia/oppia/blob/619c620d51f4af175c52d7dfe435c3dc70043f42/scripts/run_backend_tests.py#L336-L341

It looks like the error comes from trying to convert `task.exception.args[0]` to bytes. Could you print out `task.exception.args[0]` before the if-block where the code fails? That might shed some insight onto what's going on
Hey @U8NWXD, sorry for the late response. I am able to run the e2e tests locally by installing java, I guess that's why it wasn't working. But I'm still not able to run the backend tests locally. The test seems to be stuck at `18:49:44 ERROR core.platform.cache.redis_cache_services_test.RedisCacheServicesUnitTests`

<img width="1440" alt="Screenshot 2021-02-17 at 12 29 52 AM" src="https://user-images.githubusercontent.com/66058206/108109178-8f463080-70b7-11eb-90da-1dd354ff260a.png">

Can you please take a look?

@krishita30j it looks like your redis server is failing to start (that's the port 6379 message). This can happen if you have a redis server already running that's left over from a previous run that didn't shutdown properly. Can you search for any existing redis server processes and shut them down? On macOS and Linux you can use `ps -ax | grep redis` to find the processes IDs and `kill [process ID]` to kill them
> @krishita30j it looks like your redis server is failing to start (that's the port 6379 message). This can happen if you have a redis server already running that's left over from a previous run that didn't shutdown properly. Can you search for any existing redis server processes and shut them down? On macOS and Linux you can use `ps -ax | grep redis` to find the processes IDs and `kill [process ID]` to kill them

@U8NWXD FYI, In mac with M1 chips redis-server is not started on its own. It has to be started by running the command `redis-server` in a different terminal.
@U8NWXD, I killed all the redis server process but the following error shows up:

<img width="1440" alt="Screenshot 2021-02-17 at 3 46 56 PM" src="https://user-images.githubusercontent.com/66058206/108190032-6a919d80-7137-11eb-8f57-dfbf77b970f8.png">

@krishita30j @gp201 So then is the issue that the e2e tests try to start the redis servers even when the servers are already running in another tab (which is required on M1 macs)? If so, then you could temporarily change the `run_e2e_tests.py` script to not start the redis servers. In the short-term, we could add a command-line argument to the `run_e2e_tests.py` script to skip starting the redis server. Long-term, though, we should be able to install redis on M1 macs normally. From the [release notes](https://raw.githubusercontent.com/redis/redis/6.0/00-RELEASENOTES) it looks like at least 6.0.10 supports M1 macs, but we are on 6.0.5. Should we consider upgrading the redis cli?
@U8NWXD We should go ahead and implement the short term fix, since we're starting to see a few contributors with M1 chip Macs. We should also consider upgrading the redis server. That would also make it easier to run `python -m scripts.start`. Since the current method to run the cmd requires one to run the `redis-server` in a seperate terminal. [ref](https://github.com/oppia/oppia/issues/11584#issuecomment-766111198). 

@krishita30j can you check if you're able to install redis-server for python2 successfully? I remember there were some issues with that. Not sure
Upgrading the redis cli version we use *should* be easy, so I'm going to go ahead and try that first. I'm planning to have a PR up in the next few days
@shellymutu-grigg Thanks for creating this issue! Sorry we could not take a look at this earlier. I have assigned it to dev-workflow team member now, and hopefully we would find a fix for this soon. Also great job on providing us all the details.
@shellymutu-grigg, did you find any fix for this?
Although I tried setting up the oppia env using Rosetta 2, I came across a different error:

```
Failed to start server on port 6379, exiting ...
INFORMATION

Cleaning up the servers.

Cleaning up the redis_servers.
Could not connect to Redis at 127.0.0.1:6379: Connection refused
```

 (If you find any fix related to these, please let me know) 

@krishita30j @shellymutu-grigg  can u check if you can install a python package using the `pip` cmd (any package that Oppia uses). I just want to rule out the possibility that it's an architecture issue.
@gp201, yeah I can install python packages seperately using pip.
> @shellymutu-grigg, did you find any fix for this?
> Although I tried setting up the oppia env using Rosetta 2, I came across a different error:
> 
> ```
> Failed to start server on port 6379, exiting ...
> INFORMATION
> 
> Cleaning up the servers.
> 
> Cleaning up the redis_servers.
> Could not connect to Redis at 127.0.0.1:6379: Connection refused
> ```
> 
> (If you find any fix related to these, please let me know)

@krishita30j have you tried [this](https://stackoverflow.com/questions/42857551/could-not-connect-to-redis-at-127-0-0-16379-connection-refused-with-homebrew).
@DubeySandeep @shellymutu-grigg
## Findings
Given below are the findings based on the offline discussion with @krishita30j :
All findings given below were run on [Rosetta 2](https://support.apple.com/en-us/HT211861)
- The Redis package is installed using pip in the cmd line, but it does not run (was installed by oppia when the `python -m scripts.start` cmd is run).
- The Redis server is installed successfully using `pip3` (meant for python 3) cmd and also runs.
- The reason the packages don't get downloaded is because of the architecture. Not all packages are able to run on ARM architecture (M1 chips).
## Current Solution:
1. Install Rosetta 2 try [this](https://www.google.com/url?q=https://stackoverflow.com/questions/64882584/how-to-run-the-homebrew-installer-under-rosetta-2-on-m1-macbook%23:~:text%3DYou%2520can%2520run%2520Terminal%2520with,%2522Open%2520using%2520Rosetta%2522%2520option&sa=D&source=hangouts&ust=1611504651223000&usg=AFQjCNFhJysGCAFdzLswFuK5JVoxjs6CwQ)
2. Inside Rosetta perform all the installations/prerequisites.
Note: If `sudo easy_install pyyaml` does not work try using `pip3 install pyyaml`.
3. Open the rosetta terminal and download redis server using `pip3 install redis`.
4. Run redis-server with the following cmd: `redis-server` (**Don't stop the redis server**)
5. Open another rosetta terminal and run `python -m scripts.start`

EDIT: Added step-2
@sajalasati @gp201, Thanks a lot for looking into this issue, can you please confirm whether the issue is resolved now? (If yes, then I'll go ahead and add the fix to the troubleshooting wiki page and close this issue.)
@DubeySandeep The steps above were used to resolve the issue on @krishita30j computer.
@krishita30j can you cross-check the details?
> @DubeySandeep @shellymutu-grigg
> 
> ## Findings
> Given below are the findings based on the offline discussion with @krishita30j :
> All findings given below were run on [Rosetta 2](https://support.apple.com/en-us/HT211861)
> 
> * The Redis package is installed using pip in the cmd line, but it does not run (was installed by oppia when the `python -m scripts.start` cmd is run).
> * The Redis server is installed successfully using `pip3` (meant for python 3) cmd and also runs.
> * The reason the packages don't get downloaded is because of the architecture. Not all packages are able to run on ARM architecture (M1 chips).
> 
> ## Current Solution:
> 1. Install Rosetta 2 try [this](https://www.google.com/url?q=https://stackoverflow.com/questions/64882584/how-to-run-the-homebrew-installer-under-rosetta-2-on-m1-macbook%23:~:text%3DYou%2520can%2520run%2520Terminal%2520with,%2522Open%2520using%2520Rosetta%2522%2520option&sa=D&source=hangouts&ust=1611504651223000&usg=AFQjCNFhJysGCAFdzLswFuK5JVoxjs6CwQ)
> 2. Inside Rosetta perform all the installations/prerequisites.
>    Note: If `sudo easy_install pyyaml` does not work try using `brew`.
> 3. Open the rosetta terminal and download redis server using `pip3 install redis`.
> 4. Run redis-server with the following cmd: `redis-server` (**Don't stop the redis server**)
> 5. Open another rosetta terminal and run `python -m scripts.start`
> 
> EDIT: Added step-2

@gp201, I checked it again and it seems like `brew` won't work to install `pyyaml`. Instead, use `pip3 install pyyaml`. Other than this, everything else looks good.
@DubeySandeep looks like the fix is correct now. It can be added to the troubleshooting page now.
@gp201 @DubeySandeep @sajalasati while running the e2e tests locally, this error showed up (while running e2e test, I started the redis-server in another tab):

<img width="1440" alt="Screenshot 2021-01-30 at 7 27 55 PM" src="https://user-images.githubusercontent.com/66058206/106361470-0bb0e380-6344-11eb-8a91-ad2e9a564724.png">

<img width="1440" alt="Screenshot 2021-01-30 at 7 28 00 PM" src="https://user-images.githubusercontent.com/66058206/106361474-0f446a80-6344-11eb-8c80-48a28cf8f670.png">

And while running backend test, this error showed up:

<img width="1440" alt="Screenshot 2021-01-30 at 9 32 21 PM" src="https://user-images.githubusercontent.com/66058206/106361480-1ec3b380-6344-11eb-9f08-5c837ef023d1.png">

Can you please assist me in fixing this?
@U8NWXD PTAL at this issue. It is wrt to e2e test and backend test running locally in Mac with M1 chips
Hi,

I have tried these steps but receive the following error at step 2:

ERROR: Could not find a version that satisfies the requirement redis-server
ERROR: No matching distribution found for redis-server



> On 24/01/2021, at 05:10, PRANEETH <notifications@github.com> wrote:
> 
> 
> Findings
> 
> Given below are the findings based on the offline discussion with @krishita30j <https://github.com/krishita30j> :
> All findings given below were run on Rosetta 2 <https://support.apple.com/en-us/HT211861>
> The Redis package is installed using pip in the cmd line, but it does not run (was installed by oppia when the python -m scripts.start cmd is run).
> The Redis server is installed successfully using pip3 (meant for python 3) cmd and also runs.
> Current Solution:
> 
> Install Rosetta 2
> Open the rosetta terminal and download redis server using pip3 install redis-server.
> Run redis-server with the following cmd: redis-server (Don't stop the redis server)
> Open another rosetta terminal and run python -m scripts.start
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub <https://github.com/oppia/oppia/issues/11584#issuecomment-766111198>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANEUMZMBNM5KKS3DERFWX2DS3LYIZANCNFSM4VRVWSCQ>.
> 


@shellymutu-grigg can you try just executing the `scripts.run_e2e_tests_test.RunE2ETestsTests.test_work_with_non_ascii_chars` backend test? I suspect that's the one that's failing. You can run this test like so:

```console
$ python -m scripts.run_backend_tests --test_target scripts.run_e2e_tests_test.RunE2ETestsTests.test_work_with_non_ascii_chars
```
@U8NWXD sorry I should have been more clear, @shellymutu-grigg is facing a setup issue and @krishita30j is facing the e2e test and backend issue
@U8NWXD the `scripts.run_e2e_tests_test.RunE2ETestsTests.test_work_with_non_ascii_chars` test passes successfully. 
<img width="757" alt="Screenshot 2021-01-31 at 12 17 21 PM" src="https://user-images.githubusercontent.com/66058206/106377447-0fd01600-63c3-11eb-8d83-2021a82a64e0.png">


@krishita30j interesting. What about the whole `run_e2e_tests_test` module?
@U8NWXD Basic Info:

1. During setup the redis server was not running. The [current fix](https://github.com/oppia/oppia/issues/11584#issuecomment-766111198) is given above.

2. @krishita30j has already tried to run the `redis-server` in a different (rosetta 2) terminal and execute the tests in a separate (rosetta 2) terminal.
@U8NWXD running the whole `run_e2e_tests_test` module results in an ImportError.

<img width="1440" alt="Screenshot 2021-02-01 at 12 10 10 PM" src="https://user-images.githubusercontent.com/66058206/106424087-9cdca300-6487-11eb-8e36-3b5961a0bed9.png">

@krishita30j Sorry, I didn't mean running the module directly. I meant running all the backend tests in that module like this:

```console
python -m scripts.run_backend_tests --test_target scripts.run_e2e_tests_test
```
@U8NWXD, the tests pass successfully:
<img width="631" alt="Screenshot 2021-02-03 at 11 15 16 AM" src="https://user-images.githubusercontent.com/66058206/106704047-69c61b00-6611-11eb-94f3-34d176e38262.png">

Thanks @krishita30j! I looked into the code causing this error:

https://github.com/oppia/oppia/blob/619c620d51f4af175c52d7dfe435c3dc70043f42/scripts/run_backend_tests.py#L336-L341

It looks like the error comes from trying to convert `task.exception.args[0]` to bytes. Could you print out `task.exception.args[0]` before the if-block where the code fails? That might shed some insight onto what's going on
Hey @U8NWXD, sorry for the late response. I am able to run the e2e tests locally by installing java, I guess that's why it wasn't working. But I'm still not able to run the backend tests locally. The test seems to be stuck at `18:49:44 ERROR core.platform.cache.redis_cache_services_test.RedisCacheServicesUnitTests`

<img width="1440" alt="Screenshot 2021-02-17 at 12 29 52 AM" src="https://user-images.githubusercontent.com/66058206/108109178-8f463080-70b7-11eb-90da-1dd354ff260a.png">

Can you please take a look?

@krishita30j it looks like your redis server is failing to start (that's the port 6379 message). This can happen if you have a redis server already running that's left over from a previous run that didn't shutdown properly. Can you search for any existing redis server processes and shut them down? On macOS and Linux you can use `ps -ax | grep redis` to find the processes IDs and `kill [process ID]` to kill them
> @krishita30j it looks like your redis server is failing to start (that's the port 6379 message). This can happen if you have a redis server already running that's left over from a previous run that didn't shutdown properly. Can you search for any existing redis server processes and shut them down? On macOS and Linux you can use `ps -ax | grep redis` to find the processes IDs and `kill [process ID]` to kill them

@U8NWXD FYI, In mac with M1 chips redis-server is not started on its own. It has to be started by running the command `redis-server` in a different terminal.
@U8NWXD, I killed all the redis server process but the following error shows up:

<img width="1440" alt="Screenshot 2021-02-17 at 3 46 56 PM" src="https://user-images.githubusercontent.com/66058206/108190032-6a919d80-7137-11eb-8f57-dfbf77b970f8.png">

@krishita30j @gp201 So then is the issue that the e2e tests try to start the redis servers even when the servers are already running in another tab (which is required on M1 macs)? If so, then you could temporarily change the `run_e2e_tests.py` script to not start the redis servers. In the short-term, we could add a command-line argument to the `run_e2e_tests.py` script to skip starting the redis server. Long-term, though, we should be able to install redis on M1 macs normally. From the [release notes](https://raw.githubusercontent.com/redis/redis/6.0/00-RELEASENOTES) it looks like at least 6.0.10 supports M1 macs, but we are on 6.0.5. Should we consider upgrading the redis cli?
@U8NWXD We should go ahead and implement the short term fix, since we're starting to see a few contributors with M1 chip Macs. We should also consider upgrading the redis server. That would also make it easier to run `python -m scripts.start`. Since the current method to run the cmd requires one to run the `redis-server` in a seperate terminal. [ref](https://github.com/oppia/oppia/issues/11584#issuecomment-766111198). 

@krishita30j can you check if you're able to install redis-server for python2 successfully? I remember there were some issues with that. Not sure
Upgrading the redis cli version we use *should* be easy, so I'm going to go ahead and try that first. I'm planning to have a PR up in the next few days

Resolving patch:
diff --git a/scripts/common.py b/scripts/common.py
--- a/scripts/common.py
+++ b/scripts/common.py
@@ -69,10 +69,6 @@
 # the version of protobuf library being used.
 PROTOC_VERSION = PROTOBUF_VERSION
 
-# We use redis 6.0.5 instead of the latest stable build of redis (6.0.6) because
-# there is a `make test` bug in redis 6.0.6 where the solution has not been
-# released. This is explained in this issue:
-# https://github.com/redis/redis/issues/7540.
 # IMPORTANT STEPS FOR DEVELOPERS TO UPGRADE REDIS:
 # 1. Download the new version of the redis cli.
 # 2. Extract the cli in the folder that it was downloaded, most likely
@@ -88,7 +84,7 @@
 #    this message, and that all of the `make test` tests pass before you commit
 #    the upgrade to develop.
 # 7. If any tests fail, DO NOT upgrade to this newer version of the redis cli.
-REDIS_CLI_VERSION = '6.0.5'
+REDIS_CLI_VERSION = '6.0.10'
 ELASTICSEARCH_VERSION = '7.10.1'
 
 RELEASE_BRANCH_NAME_PREFIX = 'release-'


Test patch: