Repository: Exodus-Privacy/exodus-core (language: python)

Issue:
update requirements.txt and setup.py for python 3.9
The command `pip install -r requirements.txt` works for virtualenv with version 3.5. It does not with 3.9. python 3.5 is also going to be deprecated. See the warning `DEPRECATION: Python 3.5 reached the end of its life on September 13th, 2020. Please upgrade your Python as Python 3.5 is no longer maintained. pip 21.0 will drop support for Python 3.5 in January 2021. pip 21.0 will remove support for this functionality.`.

I have put the logs of the errors.

```
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DCYTHON_CLINE_IN_TRACEBACK=0 -I/usr/include/libxml2 -Isrc -Isrc/lxml/includes -I/usr/local/include/python3.9 -c src/lxml/etree.c -o build/temp.linux-x86_64-3.9/src/lxml/etree.o -w
  src/lxml/etree.c: In function ‘__Pyx_modinit_type_init_code’:
  src/lxml/etree.c:231155:32: error: ‘PyTypeObject {aka struct _typeobject}’ has no member named ‘tp_print’; did you mean ‘tp_dict’?
     __pyx_type_4lxml_5etree_Error.tp_print = 0;
                                  ^
  src/lxml/etree.c:231163:36: error: ‘PyTypeObject {aka struct _typeobject}’ has no member named ‘tp_print’; did you mean ‘tp_dict’?
     __pyx_type_4lxml_5etree_LxmlError.tp_print = 0;
```


For the moment, if you met a similar issue, you can skip with `virtualenv --python=/usr/bin/python3.5 venv-3-5` instead of `virtualenv venv -p python3` on the build instruction of the exodus core readme.md.


Discussion / hints:
Hi @counter-reverse 

Are you sure you are using python 3.9?

I don't see any reasons why the file `requirements.txt` should be linked to any version of python, and neither why you get a message about python3.5 if you are using 3.9.

I don't have any errors installing the dependencies in python 3.8.5 btw.
There is no dependency on Python.

If you own a newer Distribution you already have newer versions of the components installed. For instance on Debian Bullseye the Pillow version is 8.1.0.

<pre>apt-cache policy python3-pil
       8.1.0
         debian bullsey
         debian sid
      5.4.1
        buster</pre>

Therefore I  could solve it by increasing the version in requirements.txt. I got the solution by changing Pillow>=6.2.2 from Pillow==6.2.2.
In the file `requirements.txt` components like Pillow have dependencies which only support some versions of python:

Pillow supports these python [versions: pillow.readthedocs.io/en/stable/installation.html#python-support](https://pillow.readthedocs.io/en/stable/installation.html#python-support)

- [Debian Python 3.5 - 3.9](https://packages.debian.org/python3)
- [Ubuntu Python 3.5 - 3.9](https://packages.ubuntu.com/python3)
@pnu-s  I am confused. I posted this issue because I used to be sure that we must update the requirements.txt.

@burkhardtc Do you want to be assigned or do you prefer that I do it?

Thank you very much.
@counter-reverse Please feel free to do the right tings ;-)
I would prefer following `requirements.txt` because it works for me.

<pre>
androguard>=3.1.0
cryptography>=2.3.4
dhash>=1.3
jellyfish>=0.5.6
Pillow>=5.4.1
requests>=2.12.0
six>=1.10.0
traitlets>=4.3.2
</pre>
Same problem on Ubuntu Server 21.10 with pillow 6.2.2
`ERROR: Command errored out with exit status 1:
   command: /root/exodus-standalone/venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ra7tnic5/pillow_84aae9e847374b2c9746d3e9525956e4/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ra7tnic5/pillow_84aae9e847374b2c9746d3e9525956e4/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-k7dwqozc
       cwd: /tmp/pip-install-ra7tnic5/pillow_84aae9e847374b2c9746d3e9525956e4/
  Complete output (172 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.9
  creating build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/XbmImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImageSequence.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/XpmImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/BdfFontFile.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/GbrImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/McIdasImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImageMode.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/_util.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/Hdf5StubImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/JpegImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/GribStubImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/DcxImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/CurImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/__init__.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/GdImageFile.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/PaletteFile.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImageTk.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImageFilter.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/FtexImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/MpoImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/WebPImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/PcfFontFile.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImageColor.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/TiffImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/WmfImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImageOps.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImagePath.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImageWin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/FitsStubImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImageFile.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/PalmImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/PixarImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/BmpImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImageGrab.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/__main__.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/PngImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImageDraw2.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/FontFile.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/BlpImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/_tkinter_finder.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/MpegImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/JpegPresets.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/XVThumbImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/SgiImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/DdsImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImageChops.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImtImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/FpxImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/Jpeg2KImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/_binary.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImageCms.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImageStat.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImageEnhance.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/GimpPaletteFile.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/TiffTags.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImageMorph.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/_version.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/PyAccess.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImageDraw.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/PcdImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ExifTags.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/GimpGradientFile.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/EpsImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/PSDraw.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/FliImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/GifImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/features.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/IcoImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/IcnsImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/TgaImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/TarIO.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImageQt.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ContainerIO.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/BufrStubImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImageFont.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/IptcImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/PdfParser.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/PdfImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/SpiderImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImageMath.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/Image.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImageTransform.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImagePalette.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/PsdImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/WalImageFile.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/PcxImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/MicImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImageShow.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/SunImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/MspImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/ImImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  copying src/PIL/PpmImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
  running egg_info
  writing src/Pillow.egg-info/PKG-INFO
  writing dependency_links to src/Pillow.egg-info/dependency_links.txt
  writing top-level names to src/Pillow.egg-info/top_level.txt
  reading manifest file 'src/Pillow.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching '*.c'
  warning: no files found matching '*.h'
  warning: no files found matching '*.sh'
  warning: no previously-included files found matching '.appveyor.yml'
  warning: no previously-included files found matching '.coveragerc'
  warning: no previously-included files found matching '.codecov.yml'
  warning: no previously-included files found matching '.editorconfig'
  warning: no previously-included files found matching '.readthedocs.yml'
  warning: no previously-included files found matching 'azure-pipelines.yml'
  warning: no previously-included files matching '.git*' found anywhere in distribution
  warning: no previously-included files matching '*.pyc' found anywhere in distribution
  warning: no previously-included files matching '*.so' found anywhere in distribution
  no previously-included directories found matching '.azure-pipelines'
  no previously-included directories found matching '.travis'
  writing manifest file 'src/Pillow.egg-info/SOURCES.txt'
  running build_ext


  The headers or library files could not be found for jpeg,
  a required dependency when compiling Pillow from source.

  Please see the install instructions at:
     https://pillow.readthedocs.io/en/latest/installation.html

  Traceback (most recent call last):
    File "/tmp/pip-install-ra7tnic5/pillow_84aae9e847374b2c9746d3e9525956e4/setup.py", line 852, in <module>
      setup(
    File "/root/exodus-standalone/venv/lib/python3.9/site-packages/setuptools/__init__.py", line 162, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.9/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.9/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/root/exodus-standalone/venv/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 223, in run
      self.run_command('build')
    File "/usr/lib/python3.9/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/lib/python3.9/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/lib/python3.9/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/lib/python3.9/distutils/command/build_ext.py", line 340, in run
      self.build_extensions()
    File "/tmp/pip-install-ra7tnic5/pillow_84aae9e847374b2c9746d3e9525956e4/setup.py", line 687, in build_extensions
      raise RequiredDependencyException(f)
  __main__.RequiredDependencyException: jpeg

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-ra7tnic5/pillow_84aae9e847374b2c9746d3e9525956e4/setup.py", line 903, in <module>
      raise RequiredDependencyException(msg)
  __main__.RequiredDependencyException:

  The headers or library files could not be found for jpeg,
  a required dependency when compiling Pillow from source.

  Please see the install instructions at:
     https://pillow.readthedocs.io/en/latest/installation.html


  ----------------------------------------
  ERROR: Failed building wheel for Pillow`

Fix patch:
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index 8e8a11c..55f6e0e 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -15,12 +15,17 @@ on:
 jobs:
   build:
     runs-on: ubuntu-20.04
+    strategy:
+      matrix:
+        python-version:
+          - 3.7
+          - 3.9
     steps:
       - uses: actions/checkout@v2
-      - name: Set up Python 3.7
+      - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v2
         with:
-          python-version: 3.7
+          python-version: ${{ matrix.python-version }}
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
diff --git a/exodus_core/analysis/static_analysis.py b/exodus_core/analysis/static_analysis.py
index 1ef5238..1fb7044 100644
--- a/exodus_core/analysis/static_analysis.py
+++ b/exodus_core/analysis/static_analysis.py
@@ -1,24 +1,22 @@
-from collections import namedtuple
-from cryptography.x509.name import _SENTINEL, ObjectIdentifier, _NAMEOID_DEFAULT_TYPE, _ASN1Type, NameAttribute
-from hashlib import sha256, sha1
-from PIL import Image
-from tempfile import NamedTemporaryFile, TemporaryDirectory
 import binascii
-import dhash
 import itertools
 import logging
 import os
 import re
-import requests
-import six
 import subprocess
-# import time
 import zipfile
+from collections import namedtuple
+from future.moves import sys
+from hashlib import sha256, sha1
+from tempfile import NamedTemporaryFile, TemporaryDirectory
 
+import dhash
+import requests
+import six
 from androguard.core.bytecodes import axml
 from androguard.core.bytecodes.apk import APK
-from future.moves import sys
-# from gplaycli import gplaycli
+from cryptography.x509.name import _SENTINEL, ObjectIdentifier, _NAMEOID_DEFAULT_TYPE, _ASN1Type, NameAttribute
+from PIL import Image
 
 PHASH_SIZE = 8
 
diff --git a/requirements.txt b/requirements.txt
index 044f3d7..171d561 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -2,7 +2,7 @@ androguard==3.3.5
 cryptography==3.3.2
 dhash==1.3
 jellyfish==0.5.6
-Pillow==6.2.2
+Pillow==8.3.2
 requests==2.26.0
 six==1.15.0
 traitlets==4.3.2
diff --git a/setup.py b/setup.py
index 1b97bd5..f947158 100644
--- a/setup.py
+++ b/setup.py
@@ -41,7 +41,7 @@ def is_exe(fpath):
     'cryptography==3.3.2',
     'dhash==1.3',
     'jellyfish==0.5.6',
-    'Pillow==6.2.2',
+    'Pillow==8.3.2',
     'requests==2.26.0',
     'six==1.15.0',
     'traitlets==4.3.2'


Test patch:
diff --git a/exodus_core/analysis/test_exodus_analyze.py b/exodus_core/analysis/test_exodus_analyze.py
index 8eaad05..0e5c738 100644
--- a/exodus_core/analysis/test_exodus_analyze.py
+++ b/exodus_core/analysis/test_exodus_analyze.py
@@ -73,8 +73,8 @@ def test_icon_similarity_different_app(self):
         diff_1 = sa.get_icon_similarity(phash_1, phash_2)
         diff_2 = sa.get_icon_similarity(phash_2, phash_1)
 
-        self.assertEqual(diff_1, 0.7265625)
-        self.assertEqual(diff_2, 0.7265625)
+        self.assertEqual(diff_1, 0.734375)
+        self.assertEqual(diff_2, 0.734375)
 
     def test_icon_similarity_different_similarities(self):
         sa = StaticAnalysis()