Repository: DDMAL/CantusDB

Issue:
Errors while running test_views
The last time I tried to run our test suite, I got a bunch of errors.

Here's the traceback for two of the failing tests; the other failures are similar to one or the other of these:

```python
======================================================================
ERROR: test_provide_suggested_fulltext (main_app.tests.test_views.SourceEditChantsViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/usr/local/lib/python3.9/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.9/http/client.py", line 289, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.9/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/usr/local/lib/python3.9/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.9/http/client.py", line 289, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/code/django/cantusdb_project/main_app/tests/test_views.py", line 1780, in test_provide_suggested_fulltext
    response = self.client.get(
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 836, in get
    response = super().get(path, data=data, secure=secure, **extra)
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 424, in get
    return self.generic(
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 541, in generic
    return self.request(**r)
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 810, in request
    self.check_exception(response)
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 663, in check_exception
    raise exc_value
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 56, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/base.py", line 103, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/contrib/auth/mixins.py", line 73, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/contrib/auth/mixins.py", line 135, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/base.py", line 142, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/edit.py", line 204, in get
    return super().get(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/edit.py", line 144, in get
    return self.render_to_response(self.get_context_data())
  File "/code/django/cantusdb_project/main_app/views/chant.py", line 1665, in get_context_data
    request = requests.get(
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

======================================================================
ERROR: test_update_chant (main_app.tests.test_views.SourceEditChantsViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.9/ssl.py", line 501, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.9/ssl.py", line 1041, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.9/ssl.py", line 1310, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='cantusindex.org', port=443): Max retries exceeded with url: /json-cid/446604 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1129)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/code/django/cantusdb_project/main_app/tests/test_views.py", line 1754, in test_update_chant
    response = self.client.get(
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 836, in get
    response = super().get(path, data=data, secure=secure, **extra)
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 424, in get
    return self.generic(
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 541, in generic
    return self.request(**r)
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 810, in request
    self.check_exception(response)
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 663, in check_exception
    raise exc_value
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 56, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/base.py", line 103, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/contrib/auth/mixins.py", line 73, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/contrib/auth/mixins.py", line 135, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/base.py", line 142, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/edit.py", line 204, in get
    return super().get(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/edit.py", line 144, in get
    return self.render_to_response(self.get_context_data())
  File "/code/django/cantusdb_project/main_app/views/chant.py", line 1665, in get_context_data
    request = requests.get(
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 667, in send
    history = [resp for resp in gen]
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 667, in <listcomp>
    history = [resp for resp in gen]
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 237, in resolve_redirects
    resp = self.send(
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='cantusindex.org', port=443): Max retries exceeded with url: /json-cid/446604 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1129)')))
```

One thing that confuses me is that django is not identifying the locations in `test_views.py` where errors are occurring.

---

A few things I have tried while investigating this:
- switching from the `develop` branch to the `staging` branch and running the test suite there
- connecting to the staging server and trying to run tests there.

Locally, I get the same number of errors while on the `staging` branch as on the `develop` branch. This confuses me, since all tests were passing on that branch a couple of weeks when we last merged develop into staging for #549.

And when I try running the suite locally vs. on the staging server, I get different numbers of errors. I don't understand why this should be the case.
- Testing locally:
  - on the `develop` branch: 8 errors
  - on the `staging` branch: 8 errors
- `ssh`ing into http://206.12.88.113:
  - on the `develop` branch: 111 errors
  - on the `staging` branch: all tests pass

Investigation is ongoing! I'll post more details here as they emerge. If anyone has any ideas of what might be going on, or of things worth trying, don't hesitate to let me know!


Discussion:
Some of the errors above suggest that something may have changed on cantusindex.org, causing some of the tests to break.

When I try to visit https://cantusindex.org/json-cid/446604 in my browser (suggested by the last line of the traceback posted above), I get a page that loads properly, consisting of an empty list. `curl`ing the url seems to work too, at least most of the time:
```
~ curl https://cantusindex.org/json-cid/446604
curl: (35) error:02FFF036:system library:func(4095):Connection reset by peer
~ curl https://cantusindex.org/json-cid/446604
﻿[]%                                                                           
~ curl https://cantusindex.org/json-cid/446604
﻿[]%   
```
occasionally, we get similar error messages, but for `/json-con/` urls as well:

```
======================================================================
ERROR: test_url_and_templates (main_app.tests.test_views.ChantDetailViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.9/ssl.py", line 501, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.9/ssl.py", line 1041, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.9/ssl.py", line 1310, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='cantusindex.org', port=443): Max retries exceeded with url: /json-con/014866 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1129)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/code/django/cantusdb_project/main_app/tests/test_views.py", line 530, in test_url_and_templates
    response = self.client.get(reverse("chant-detail", args=[chant.id]))
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 836, in get
    response = super().get(path, data=data, secure=secure, **extra)
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 424, in get
    return self.generic(
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 541, in generic
    return self.request(**r)
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 810, in request
    self.check_exception(response)
  File "/usr/local/lib/python3.9/site-packages/django/test/client.py", line 663, in check_exception
    raise exc_value
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 56, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/base.py", line 103, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/base.py", line 142, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/generic/detail.py", line 109, in get
    context = self.get_context_data(object=self.object)
  File "/code/django/cantusdb_project/main_app/views/chant.py", line 112, in get_context_data
    response = requests.get(
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 667, in send
    history = [resp for resp in gen]
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 667, in <listcomp>
    history = [resp for resp in gen]
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 237, in resolve_redirects
    resp = self.send(
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='cantusindex.org', port=443): Max retries exceeded with url: /json-con/014866 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1129)')))
```
Applying a label of "urgent", because we should really get this fixed before this year's training week, for the benefit of anyone just getting started working on CantusDB.
Detective work:
- locally, `test_functions` (contains only 3 tests) runs fine.
- locally, `test_models` (35 tests) also runs fine
The plot thickens! I'm going through the test suite, commenting out tests. I run the test suite - 1 error. Comment out the function causing the error, and run the test suite again - 1 error, but on a different test! Run the test suite again, without making further changes - 2 errors this time!

One of the tracebacks mentioned the python `threading` library, which we only use (afaik) when fetching lists of concordances from Cantus Index. My best guess at the moment is that calls we're making to CI's/others' APIs are leading to responses that are wonky in one way or another, in different ways from one request to the next, leading to this unpredictability.
Okay, something is up with Cantus Index:
```shell
$ curl cantusindex.org/json-con/703437
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://cantusindex.org/json-con/703437">here</a>.</p>
</body></html>
```

Further,
```shell
$ curl https://cantusindex.org/json-con/703437
﻿[]%                                                                           
$ curl http://cantusindex.org/json-con/703437
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://cantusindex.org/json-con/703437">here</a>.</p>
</body></html>
```
Which strongly suggests that CantusIndex is simply returning 301 errors whenever a request is made to `http...` rather than to `https...`. This hypothesis does not explain the non-determinism of the test failures, however.
I went through the views and replaced `http` with `https` wherever I could. I also updated the version of `urllib3` from `1.26.5` to `1.26.15` in `requirements.txt`. We've gone from 8 or 9 errors down to 4 or 5.
@dchiller and I spent some time troubleshooting this today. We're still stumped, but we discovered a couple of things that are potentially helpful:
- we are, in fact, getting line numbers within `test_views.py` - see, for example, the following, hidden in the middle of one of the tracebacks:
```python
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/code/django/cantusdb_project/main_app/tests/test_views.py", line 1780, in test_provide_suggested_fulltext
    response = self.client.get(
```
- the above suggests that something is going wrong in Django's handling of requests/responses, and not with the tests themselves
- While some of the affected views depend on external resources (e.g. Cantus Index's APIs), we found at least one view where errors are occurring where there are no requests made to external resources. We ran just this test 5 times, and it generated an error 4 times
- I ran the full test suite a half dozen times, and kept track of which tests failed each time. None of the tests led to an error every time - each one worked at least once. Every time I ran the test suite, a new test failed that hadn't previously.

Further thoughts: so yesterday, while @dchiller and I were pairing on this, we switched to the `staging` branch (for which all the tests were certainly working the last time we merged `develop` into `staging`), and ran the test suite several times - we observed the same sort of errors outlined in this issue's first few comments. Just now, I ran the test suite two times on the staging server (i.e. on the `staging` branch), and two times on the production server (on the `production` branch). All four times, all tests passed. This suggests that I might have messed up my local environment somehow. I might try uninstalling and reinstalling the project and related apps (i.e. Docker) to see if that helps.
In order to ensure these errors did not have anything to do with some kind of corruption in @jacobdgm's local environment, I built the CantusDB containers locally from the `develop` branch. 

Running the views tests twice resulted in different tests failing (for the same `ConnectionError` as mentioned above) each time. 
I've now found multiple tests that (seemingly randomly) succeed, fail with a `ConnectionError,` and fail (much less frequently) with a `SSLError`. For reference, the two tests that I've tried thus far where this seems to be true (`main_app.tests.test_views.SourceEditChantsViewTest.test_update_chant` and `main_app.tests.test_views.SourceEditChantsViewTest.test_provide_suggested_fulltext`)

It seems possible that the same thing is at the root of errors, but for some reason, the error caught is not deterministic? On the one hand, this seems helpful as the `SSLError` is more informative (?). 
Update: after having sent an email to Jan, it appears that nothing has changed on CI's end lately, but that he's come across this sort of problem from time to time and it just needs to be worked around. Some research suggests we can use Python `unittest.Mock` objects to substitute known return values for values coming from external resources, which will let us isolate our tests from whatever is going on at CantusIndex, elsewhere on the internet, etc.

Noting that I'm occasionally getting an SSLError when I try to visit a chant detail page locally, so this has the potential to extend beyond tests: ![Screenshot 2023-04-27 at 11 50 03 AM](https://user-images.githubusercontent.com/58090591/234916883-3fcff4a7-4e73-4650-a854-38164da4d72b.png)

I got this while working on adding `timeout=5` to our `requests.get` calls. We only get this error sometimes, and when we do, we get the error page almost instantaneously (rather than after 5 seconds).

I don't recall coming across this anywhere while testing our production or staging servers, so it seems to be a problem only when developing locally. I'll keep my eyes open in the future for this, though.

Resolving patch:
diff --git a/django/cantusdb_project/main_app/views/chant.py b/django/cantusdb_project/main_app/views/chant.py
--- a/django/cantusdb_project/main_app/views/chant.py
+++ b/django/cantusdb_project/main_app/views/chant.py
@@ -361,9 +361,9 @@ def get_context_data(self, **kwargs):
             t = threading.Thread(
                 target=requests.get,
                 args={
-                    "url":f"https://cantusindex.org/json-con/{chant.cantus_id}/refresh",
-                    "timeout":5,
-                }
+                    "url": f"https://cantusindex.org/json-con/{chant.cantus_id}/refresh",
+                    "timeout": 5,
+                },
             )
             t.start()
 
@@ -1633,7 +1633,7 @@ def get_chants_with_folios(chants_in_feast):
             cantus_id = current_chant.cantus_id
 
             request = requests.get(
-                "https://cantusindex.org/json-cid/{cantus_id}",
+                f"https://cantusindex.org/json-cid/{cantus_id}",
                 timeout=5,
             )
             request_text = json.loads(request.text[2:])


Test patch: