Repository: fidals/shopelectro

Issue:
502 error on category pages
We can reproduce it with code below.

Seems it's connected with https://github.com/fidals/shopelectro/issues/648, but it's not checked yet

```
import requests

BASE = 'https://www.shopelectro.ru'

broken_urls = []
for c in CategoryPage.objects.all():
	response = requests.get(BASE + c.url)
	if response.status_code != 200:
		broken_urls += (response.status_code, response.url)

print('\n'.join((str(url) for url in broken_urls)))

---

# output:
502
https://www.shopelectro.ru/catalog/categories/batarei-saliutov-moshchnye/
502
https://www.shopelectro.ru/catalog/categories/khlopushki-pnevmaticheskie/
502
https://www.shopelectro.ru/catalog/categories/izmeritelnye-pribory-211/
502
https://www.shopelectro.ru/catalog/categories/dlia-tiporazmera-aaaaa/
502
https://www.shopelectro.ru/catalog/categories/tiporazmer-aa/
502
https://www.shopelectro.ru/catalog/categories/tiporazmer-6f22/
502
https://www.shopelectro.ru/catalog/categories/invertory/
502
https://www.shopelectro.ru/catalog/categories/palchikovye-aa-r6-lr6-fr6-316/
502
https://www.shopelectro.ru/catalog/categories/mizinchikovye-aaa-r03-lr03-fr03-286/
```


Discussion:
I believe it is related to a slow response time and/or wrong response timeouts
```python
import requests
BASE = 'https://www.shopelectro.ru'
broken_urls = []
ok_after_cache = []
for c in CategoryPage.objects.all()[10:60]:
    response = requests.get(BASE + c.url)
    if response.status_code != 200:
        broken_urls.append((response.status_code, response.url))
        response = requests.get(BASE + c.url)
        if response.status_code == 200: ok_after_cache.append((response.status_code, response.url))
print(list(zip(ok_after_cache, broken_urls)))
```
```
[((200,
   'https://www.shopelectro.ru/catalog/categories/kombinirovannye-saliuty/'),
  (502,
   'https://www.shopelectro.ru/catalog/categories/kombinirovannye-saliuty/')),
 ((200,
   'https://www.shopelectro.ru/catalog/categories/kombinirovannye-saliuty-moshchnye/'),
  (502,
   'https://www.shopelectro.ru/catalog/categories/kombinirovannye-saliuty-moshchnye/')),
 ((200, 'https://www.shopelectro.ru/catalog/categories/rimskie-svechi/'),
  (502, 'https://www.shopelectro.ru/catalog/categories/rimskie-svechi/')),
 ((200, 'https://www.shopelectro.ru/catalog/categories/petardy/'),
  (502, 'https://www.shopelectro.ru/catalog/categories/petardy/')),
 ((200,
   'https://www.shopelectro.ru/catalog/categories/dlia-tiporazmera-aaaaa/'),
  (502,
   'https://www.shopelectro.ru/catalog/categories/dlia-tiporazmera-aaaaa/')),
 ((200, 'https://www.shopelectro.ru/catalog/categories/dlia-fotoapparatov/'),
  (502, 'https://www.shopelectro.ru/catalog/categories/dlia-fotoapparatov/')),
 ((200,
   'https://www.shopelectro.ru/catalog/categories/dlia-akkumuliatornykh-sborok-1-2-v-12-v/'),
  (502,
   'https://www.shopelectro.ru/catalog/categories/dlia-akkumuliatornykh-sborok-1-2-v-12-v/')),
 ((200,
   'https://www.shopelectro.ru/catalog/categories/setevye-perekhodniki-i-razvetviteli-224/'),
  (502,
   'https://www.shopelectro.ru/catalog/categories/setevye-perekhodniki-i-razvetviteli-224/')),
 ((200,
   'https://www.shopelectro.ru/catalog/categories/elektrodinamicheskie-fonari/'),
  (502,
   'https://www.shopelectro.ru/catalog/categories/elektrodinamicheskie-fonari/'))]
```
Python 3.7 seems to be involved in this

Caught [these logs](https://gist.github.com/artemiy312/f1070920346e6a2e2ff20f070f44f02f) on prod.

Let's try to rollback to 3.6 and fix this issue later
@artemiy312 , it's good idea
@artemiy312 , it's also maybe because lack of memory on prod. It's part of `top` out:
```
KiB Mem:  16436468 total, 16136492 used,
KiB Swap:  1048572 total,  1048448 used,
```
@duker33 I will increase swap and log here the result
@artemiy312 , seems our celery leaks by memory.
It's for separated task:
```
# top + 'shift+m'

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 8332 root      20   0 4170488 3,727g   3204 S   0,3 23,8  43:32.82 celery
11042 root      20   0 2182448 1,907g   5320 S   0,0 12,2  66:08.94 celery
 8689 root      20   0 2035744 1,765g   3304 S   0,3 11,3  40:38.39 celery
 8361 root      20   0 2146972 1,514g   3136 S   0,3  9,7  35:19.81 celery
 8821 root      20   0  529992 345720   6204 S   0,0  2,1 344:18.45 celery
 8820 root      20   0  466764 283088   4156 S   0,0  1,7 342:56.93 celery
29743 root      20   0  400840 233340   8976 S   0,0  1,4   2:23.28 gunicorn
11139 root      20   0  380736 213260   9052 S   0,0  1,3   1:09.48 gunicorn
13233 root      20   0  377800 210280   9052 S   0,0  1,3   1:17.39 gunicorn
25608 root      20   0  376860 209224   9052 S   0,0  1,3   1:24.41 gunicorn
26918 root      20   0  373456 206060   8988 S   0,0  1,3   3:20.46 gunicorn

```
I have restarted the worker, but it didn't help. 502 error still happens   
@duker33 
>leaks by memory.

It may be related to python 3.7 too, because celery still doesn't support this version

Resolving patch:
diff --git a/shopelectro/celery.py b/shopelectro/celery.py
--- a/shopelectro/celery.py
+++ b/shopelectro/celery.py
@@ -83,11 +83,11 @@
     task_default_queue='default',
     task_default_exchange='default',
     task_default_routing_key='default',
-    task_ignore_result=True,
-    worker_pool_restarts=1000,
+    worker_pool_restarts=True,
     task_routes=task_routes,
     task_queues=task_queues,
     beat_schedule=beat_schedule,
+    worker_max_memory_per_child=200000,  # after 250MB will restart
 )
 
 # Load task modules from all registered Django app configs.


Test patch: