Repository: fossasia/open-event-server

Issue:
Image resizing is not working
https://github.com/fossasia/open-event-server/blob/9bb9f68305c261713ee486f3840ebcf55dc70d23/app/api/helpers/files.py#L80

urlib is throwing 403 Forbidden error for almost all images. 

Please use requests and verify image resizing works. Also, create a management command which finds all events which have no resized images and resizes them


Discussion:
@codedsun Can you please take this up?
Taking this up
@iamareebjamal I learnt that this task creates resized image and stores in static folder. How do i test that for 403 error via which api `upload image?` Please guide me how should I proceed?

As when I am hitting through the frontend and selecting the image like event background, event logo it works fine. 

And for management command do I need to add the task in the celery?
Send this as original image URL in Postman - https://storage.eventyay.com/eventyay.com/events/758/background/Vk0zSTdyZX/1517e947-29df-4a3c-9580-373db3ba9ba9.png

Also, working fine means the image is resized and all image URLs are correctly set. Is that happening?

For management command, no
@iamareebjamal 

![image](https://user-images.githubusercontent.com/20026440/69666656-9c41c300-10b2-11ea-8871-edccba04100e.png)

They are null
@iamareebjamal When the event is created the files.py function is run. What is expected that large_image_url, thumbnail_image_url - should be created with a valid url and saved in the db? Am I right?
Yes
@iamareebjamal Okay I will debug it tomorrow.
Status? PATCH endpoint is not as important. That is atleast working with its limitation. This is completely broken
@iamareebjamal Sorry I missed that due to health issues. I am resuming and working on it now.
@iamareebjamal I am finding a way to check that celery worker is running or not on my machine also how do I turn on logs for celery.. I see that function is called, but I m debugging more about it.
celery won't run if you've not run it. It's simple
Did you follow the instructions and run celery? https://github.com/fossasia/open-event-server/blob/development/docs/installation/basic.md
Yeah, while running `celery worker -A app.celery`

I am having error
```
[2019-12-05 17:39:30,349: WARNING/ForkPoolWorker-3] HEAD http://localhost:9200/event [status:N/A request:0.004s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 159, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 80, in create_connection
    raise err
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 70, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 238, in perform_request
    method, url, body, retries=Retry(False), headers=request_headers, **kw
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 344, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 181, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 168, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x10db7e750>: Failed to establish a new connection: [Errno 61] Connection refused
```
ENABLE_ELASTICSEARCH=False in your .env
@iamareebjamal For the locally saved images on the server it's working fine for me.i.e. From frontend I tried to upload the images, they are getting resized and stored in the events table. 

Now as per the requirement, that creating an event by using a postman request with imageurl of not a locally saved file, it's throwing 403 error. I am debugging it now.
No need to debug. The cause and solution both are written in the issue itself
@iamareebjamal Lol, what should I do then? 
Follow what's written in the issue
>Also, create a management command which finds all events which have no resized images and resizes them?

How do we write a management command, please guide me? 
Also, What should be done for 403 error?


What i understood is that, the image can't be accessed by urllib but that's right as from fronted while uploading images we are uploading that to db and returning a locally saved file URL.

> urlib is throwing 403 Forbidden error for almost all images. 
> 
> Please use requests and verify image resizing works.

I have literally pasted what I wrote in the issue. Now tell me what you don't understand
>No need to debug. The cause and solution both are written in the issue itself

What's the solution here? I need to fix 403 error of urllib

>urlib is throwing 403 Forbidden error for almost all images.

It's throwing for the files which are not saved locally on the server, do I need to fix this for the images which are from the source other than local machine or test for which are locally saved on the server?

Also when I am uploading the images from the frontend , it's working fine. They are not working for images not saved on local machine


Sorry for asking such question but help me where I am not able to understand. Thanks
> Please use [requests](https://github.com/psf/requests) and verify image resizing works.
That hyperlink made a ton of difference for a beginner like me. I thought request as a hitting api.

>Please use requests and verify image resizing works.

For this you meant to replace using urllib and use requests instead to open the file and then perform the resizing. Am I correct here?
Yes
@iamareebjamal If that hyperlink would have been present, it wouldn't have taken a day. Thanks :) 

Resolving patch:
diff --git a/app/api/helpers/tasks.py b/app/api/helpers/tasks.py
--- a/app/api/helpers/tasks.py
+++ b/app/api/helpers/tasks.py
@@ -135,7 +135,7 @@ def resize_event_images_task(self, event_id, original_image_url):
         event.icon_image_url = uploaded_images['icon_image_url']
         save_to_db(event)
         logging.info('Resized images saved successfully for event with id: {}'.format(event_id))
-    except (urllib.error.HTTPError, urllib.error.URLError):
+    except (requests.exceptions.HTTPError, requests.exceptions.URLError):
         logging.exception('Error encountered while generating resized images for event with id: {}'.format(event_id))
 
 
@@ -152,7 +152,7 @@ def resize_user_images_task(self, user_id, original_image_url):
         user.icon_image_url = uploaded_images['icon_image_url']
         save_to_db(user)
         logging.info('Resized images saved successfully for user with id: {}'.format(user_id))
-    except (urllib.error.HTTPError, urllib.error.URLError):
+    except (requests.exceptions.HTTPError, requests.exceptions.InvalidURL):
         logging.exception('Error encountered while generating resized images for user with id: {}'.format(user_id))
 
 
@@ -166,7 +166,7 @@ def sponsor_logos_url_task(self, event_id):
             sponsor.logo_url = new_logo_url
             save_to_db(sponsor)
             logging.info('Sponsor logo url successfully generated')
-        except(urllib.error.HTTPError, urllib.error.URLError):
+        except(requests.exceptions.HTTPError, requests.exceptions.InvalidURL):
             logging.exception('Error encountered while logo generation')
 
 
@@ -181,7 +181,7 @@ def resize_speaker_images_task(self, speaker_id, photo_url):
         speaker.icon_image_url = uploaded_images['icon_image_url']
         save_to_db(speaker)
         logging.info('Resized images saved successfully for speaker with id: {}'.format(speaker_id))
-    except (urllib.error.HTTPError, urllib.error.URLError):
+    except (requests.exceptions.HTTPError, requests.exceptions.InvalidURL):
         logging.exception('Error encountered while generating resized images for event with id: {}'.format(speaker_id))
 
 
diff --git a/manage.py b/manage.py
--- a/manage.py
+++ b/manage.py
@@ -10,8 +10,10 @@
 from populate_db import populate
 from flask_migrate import stamp
 from sqlalchemy.engine import reflection
-
+from sqlalchemy import or_
 from tests.all.integration.auth_helper import create_super_admin
+from app.api.helpers.tasks import resize_event_images_task
+from app.api.helpers.tasks import resize_speaker_images_task
 
 
 @manager.command
@@ -37,6 +39,21 @@ def add_event_identifier():
         save_to_db(event)
 
 
+@manager.command
+def fix_event_and_speaker_images():
+    events = Event.query.filter(Event.original_image_url.isnot(None),
+                                or_(Event.thumbnail_image_url == None, Event.large_image_url == None,
+                                    Event.icon_image_url == None)).all()
+    for event in events:
+        resize_event_images_task.delay(event.id, event.original_image_url)
+
+    speakers = Speaker.query.filter(Speaker.photo_url.isnot(None),
+                                    or_(Speaker.icon_image_url == None,
+                                        Speaker.small_image_url == None, Speaker.thumbnail_image_url == None))
+    for speaker in speakers:
+        resize_speaker_images_task.delay(speaker.id, speaker.photo_url)
+
+
 @manager.command
 def fix_digit_identifier():
     events = Event.query.filter(Event.identifier.op('~')('^[0-9\.]+$')).all()


Test patch: