Repository: fossasia/open-event-server

Issue:
Application error when open admin/ page
> 2016-08-15T08:27:00.294911+00:00 app[web.1]: [2016-08-15 08:27:00 +0000] [17] [ERROR] Error handling request /socket.io/?EIO=3&transport=websocket&sid=78177a028b234f31984e345e4fac5ca3
> 2016-08-15T08:27:00.294929+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/workers/async.py", line 52, in handle
> 2016-08-15T08:27:00.294929+00:00 app[web.1]:     self.handle_request(listener_name, req, client, addr)
> 2016-08-15T08:27:00.294932+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/http/wsgi.py", line 418, in close
> 2016-08-15T08:27:00.294931+00:00 app[web.1]:     resp.close()
> 2016-08-15T08:27:00.294930+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/workers/async.py", line 112, in handle_request
> 2016-08-15T08:27:00.294933+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/http/wsgi.py", line 334, in send_headers
> 2016-08-15T08:27:00.294932+00:00 app[web.1]:     self.send_headers()
> 2016-08-15T08:27:00.294933+00:00 app[web.1]:     tosend = self.default_headers()
> 2016-08-15T08:27:00.294934+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/http/wsgi.py", line 315, in default_headers
> 2016-08-15T08:27:00.294935+00:00 app[web.1]:     elif self.should_close():
> 2016-08-15T08:27:00.294936+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/http/wsgi.py", line 238, in should_close
> 2016-08-15T08:27:00.294936+00:00 app[web.1]:     if self.status_code < 200 or self.status_code in (204, 304):
> 2016-08-15T08:27:00.294937+00:00 app[web.1]: AttributeError: 'Response' object has no attribute 'status_code'



Discussion:
@rafalkowalski If you mean the AttributeError in Gunicorn it shouldn't be an issue. It's because Gunicorn doesn't understand WebSocket requests. Here's more from a similar issue at Flask-SocketIO:

https://github.com/miguelgrinberg/Flask-SocketIO/issues/223

It doesn't block any communication or stop the server.

@shivamMg Have you visited /admin page? http://open-event-dev.herokuapp.com/admin/ maybe you know why this page can't be open?

@shivamMg this is the error log:

> 2016-08-15T20:44:29.679383+00:00 heroku[web.1]: Process running mem=524M(102.4%)
> 2016-08-15T20:44:29.679531+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
> 2016-08-15T20:44:50.925820+00:00 heroku[web.1]: Process running mem=534M(104.3%)
> 2016-08-15T20:44:50.925894+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
> 2016-08-15T20:44:38+00:00 app[heroku-redis]: source=HEROKU_REDIS_BLUE sample#active-connections=1 sample#load-avg-1m=0.13 sample#load-avg-5m=0.08 sample#load-avg-15m=0.075 sample#read-iops=0 sample#write-iops=0 sample#memory-total=15664884.0kB sample#memory-free=12355324.0kB sample#memory-cached=435956kB sample#memory-redis=333472bytes sample#hit-rate=0.41841 sample#evicted-keys=0
> 2016-08-15T20:44:43+00:00 app[heroku-redis]: source=REDIS sample#active-connections=12 sample#load-avg-1m=0.06 sample#load-avg-5m=0.045 sample#load-avg-15m=0.05 sample#read-iops=0 sample#write-iops=0 sample#memory-total=15664884.0kB sample#memory-free=13732316.0kB sample#memory-cached=417796kB sample#memory-redis=535808bytes sample#hit-rate=0.34037 sample#evicted-keys=0
> 2016-08-15T20:45:12.084207+00:00 heroku[web.1]: Process running mem=541M(105.8%)
> 2016-08-15T20:45:12.084207+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
> 2016-08-15T20:45:17.535920+00:00 app[web.1]: Fantastico Concerto
> 2016-08-15T20:45:17.457146+00:00 app[web.1]: BAWHHH GAWWWD
> 2016-08-15T20:45:17.688250+00:00 app[web.1]: PayPal Payment Test Event 
> 2016-08-15T20:45:17.612078+00:00 app[web.1]: sddfs
> 2016-08-15T20:45:17.750381+00:00 app[web.1]: PayPal and Stripe Payment Test Event
> 2016-08-15T20:45:17.835682+00:00 app[web.1]: Cool HK event
> 2016-08-15T20:45:17.910709+00:00 app[web.1]: Some Event
> 2016-08-15T20:45:17.989760+00:00 app[web.1]: Stripe Payment Test Event
> 2016-08-15T20:45:18.118415+00:00 app[web.1]: Aayush
> 2016-08-15T20:45:18.257221+00:00 app[web.1]: jaja
> 2016-08-15T20:45:32.880223+00:00 heroku[web.1]: Process running mem=548M(106.9%)
> 2016-08-15T20:45:32.880360+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
> 2016-08-15T20:45:54.183378+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
> 2016-08-15T20:45:54.183260+00:00 heroku[web.1]: Process running mem=548M(106.9%)
> 2016-08-15T20:45:57+00:00 app[heroku-redis]: source=HEROKU_REDIS_BLUE sample#active-connections=1 sample#load-avg-1m=0.06 sample#load-avg-5m=0.07 sample#load-avg-15m=0.07 sample#read-iops=0 sample#write-iops=0 sample#memory-total=15664884.0kB sample#memory-free=12355040.0kB sample#memory-cached=435956kB sample#memory-redis=333472bytes sample#hit-rate=0.41841 sample#evicted-keys=0
> 2016-08-15T20:46:14.889039+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
> 2016-08-15T20:46:14.888944+00:00 heroku[web.1]: Process running mem=548M(106.9%)
> 2016-08-15T20:46:03+00:00 app[heroku-redis]: source=REDIS sample#active-connections=12 sample#load-avg-1m=0.04 sample#load-avg-5m=0.045 sample#load-avg-15m=0.05 sample#read-iops=0 sample#write-iops=0 sample#memory-total=15664884.0kB sample#memory-free=13732556.0kB sample#memory-cached=417796kB sample#memory-redis=535808bytes sample#hit-rate=0.34037 sample#evicted-keys=0

I did a little searching on the error.. It's a problem of heroku kindof..... Due to some reason the memory quota to be used is exceeding.... It can be due to various reasons.. Some of the reasons I found out was too many migrations.... Too much memory used in runtime due to too many service workers....
but this error seems to arrive mainly for ruby on rails or NodeJS codes... no such issue reported for flask...
So I am a little confused what is the exact reason for this happening..
And then in a blog I found this as well:
I’ve been pulling my hair out wondering why my video processing worker dynos are getting R14 “Memory Quota” exceeded warnings, and only after I instrumented my code to take memory snapshots (ps -ef + cat /proc/meminfo at relevant timings) I got to suspecting that buffer cache is counted against me in my memory quota (why isn’t this documented? why can’t Heroku’s support resolve my ticket asking for explanations regarding my ps -ef outputs for a few weeks now?). We get false R14 positives all the time.
https://tech.blog.aknin.name/2012/03/09/heroku-is-great-however/
This is the only link I found in heroku about R14 https://devcenter.heroku.com/articles/error-codes#r14-memory-quota-exceeded

@aviaryan @niranjan94 any idea?


Resolving patch:
diff --git a/app/helpers/data_getter.py b/app/helpers/data_getter.py
--- a/app/helpers/data_getter.py
+++ b/app/helpers/data_getter.py
@@ -41,7 +41,7 @@
 from ..models.order import Order
 from .language_list import LANGUAGE_LIST
 from .static import EVENT_TOPICS, EVENT_LICENCES, PAYMENT_COUNTRIES, PAYMENT_CURRENCIES, DEFAULT_EVENT_IMAGES
-from app.helpers.helpers import get_event_id, string_empty, represents_int
+from app.helpers.helpers import get_event_id, string_empty, represents_int, get_count
 from flask.ext import login
 from flask import flash, abort
 import datetime
@@ -749,16 +749,17 @@ def get_expired_orders():
 
     @staticmethod
     def get_all_super_admins():
-        return len(User.query.filter_by(is_super_admin=True).all())
+        return get_count(User.query.filter_by(is_super_admin=True))
 
     @staticmethod
     def get_all_admins():
-        return len(User.query.filter_by(is_admin=True).all())
+        return get_count(User.query.filter_by(is_admin=True))
 
     @staticmethod
     def get_all_registered_users():
-        return len(User.query.filter_by(is_verified=True).all())
+        return get_count(User.query.filter_by(is_verified=True))
 
+    # TODO Make this more efficient
     @staticmethod
     def get_all_organizers():
         events = Event.query.all()
@@ -771,6 +772,7 @@ def get_all_organizers():
 
         return len(organizers)
 
+    # TODO Make this more efficient
     @staticmethod
     def get_all_co_organizers():
         events = Event.query.all()
@@ -783,6 +785,7 @@ def get_all_co_organizers():
 
         return len(co_organizers)
 
+    # TODO Make this more efficient
     @staticmethod
     def get_all_track_organizers():
         events = Event.query.all()
@@ -809,28 +812,28 @@ def get_all_attendees():
 
     @staticmethod
     def get_all_accepted_sessions():
-        return len(Session.query.filter_by(state='accepted').all())
+        return get_count(Session.query.filter_by(state='accepted'))
 
     @staticmethod
     def get_all_rejected_sessions():
-        return len(Session.query.filter_by(state='rejected').all())
+        return get_count(Session.query.filter_by(state='rejected'))
 
     @staticmethod
     def get_all_draft_sessions():
-        return len(Session.query.filter_by(state='pending').all())
+        return get_count(Session.query.filter_by(state='pending'))
 
     @staticmethod
     def get_email_by_times():
         email_times = []
-        email_in_last_24 = len(
-            Mail.query.filter(datetime.datetime.now() - Mail.time <= datetime.timedelta(hours=24)).all())
-        email_in_last_3_days = len(
-            Mail.query.filter(datetime.datetime.now() - Mail.time <= datetime.timedelta(days=3)).all())
-        email_in_last_7_days = len(
-            Mail.query.filter(datetime.datetime.now() - Mail.time <= datetime.timedelta(days=7)).all())
-        email_in_last_30_days = len(
-            Mail.query.filter(datetime.datetime.now() - Mail.time <= datetime.timedelta(days=30)).all())
-        total_emails = len(Mail.query.all())
+        email_in_last_24 = get_count(
+            Mail.query.filter(datetime.datetime.now() - Mail.time <= datetime.timedelta(hours=24)))
+        email_in_last_3_days = get_count(
+            Mail.query.filter(datetime.datetime.now() - Mail.time <= datetime.timedelta(days=3)))
+        email_in_last_7_days = get_count(
+            Mail.query.filter(datetime.datetime.now() - Mail.time <= datetime.timedelta(days=7)))
+        email_in_last_30_days = get_count(
+            Mail.query.filter(datetime.datetime.now() - Mail.time <= datetime.timedelta(days=30)))
+        total_emails = get_count(Mail.query)
 
         email_times.append(email_in_last_24)
         email_times.append(email_in_last_3_days)
diff --git a/app/views/admin/super_admin/super_admin.py b/app/views/admin/super_admin/super_admin.py
--- a/app/views/admin/super_admin/super_admin.py
+++ b/app/views/admin/super_admin/super_admin.py
@@ -2,7 +2,7 @@
 
 from app.views.admin.super_admin.super_admin_base import SuperAdminBaseView
 from ....helpers.data_getter import DataGetter
-from app.helpers.helpers import get_latest_heroku_release, get_commit_info
+from app.helpers.helpers import get_latest_heroku_release, get_commit_info, get_count
 
 
 class SuperAdminView(SuperAdminBaseView):
@@ -10,16 +10,17 @@ class SuperAdminView(SuperAdminBaseView):
     @expose('/')
     def index_view(self):
         events = DataGetter.get_all_events()[:5]
-        number_live_events = DataGetter.get_all_live_events().count()
-        number_draft_events = DataGetter.get_all_draft_events().count()
-        number_past_events = DataGetter.get_all_past_events().count()
+        number_live_events = get_count(DataGetter.get_all_live_events())
+        number_draft_events = get_count(DataGetter.get_all_draft_events())
+        number_past_events = get_count(DataGetter.get_all_past_events())
         super_admins = DataGetter.get_all_super_admins()
         admins = DataGetter.get_all_admins()
         registered_users = DataGetter.get_all_registered_users()
-        organizers = DataGetter.get_all_organizers()
-        co_organizers = DataGetter.get_all_co_organizers()
-        track_organizers = DataGetter.get_all_track_organizers()
-        attendees= DataGetter.get_all_attendees()
+        # TODO Fix function and correct this
+        organizers = 0
+        co_organizers = 0
+        track_organizers = 0
+        attendees = 0
         accepted_sessions = DataGetter.get_all_accepted_sessions()
         rejected_sessions = DataGetter.get_all_rejected_sessions()
         draft_sessions = DataGetter.get_all_draft_sessions()


Test patch: