Repository: freedomofpress/securedrop

Issue:
Debian package securedrop-app-code is not getting upgraded
# Bug

## Description

In my prod vm `securedrop-app-code` package is not getting upgraded.

## Steps to Reproduce

Follow the QA steps to fetch the latest packages from apt-test repo.

## Expected Behavior

We should get the 0.7~RC1 packages.

## Actual Behavior

```
$ dpkg -al | grep secure
ii  openssh-client                       1:6.6p1-2ubuntu2.10                        amd64        secure shell (SSH) client, for secure access to remote machines
ii  openssh-server                       1:6.6p1-2ubuntu2.10                        amd64        secure shell (SSH) server, for secure access from remote machines
ii  openssh-sftp-server                  1:6.6p1-2ubuntu2.10                        amd64        secure shell (SSH) sftp server module, for SFTP access from remote machines
ii  secure-delete                        3.1-6                                      amd64        tools to wipe files, free disk space, swap and memory
ii  securedrop-app-code                  0.6                                        amd64        Packages the SecureDrop application code pip dependencies and apparmor profiles. This package will put the apparmor profiles in enforce mode. This package does use pip to install the pip wheelhouse
ii  securedrop-config                    0.1.1+0.7.0~rc1                            all          Establishes baseline system state for running SecureDrop.
ii  securedrop-grsec                     4.4.115+r1                                 amd64        Metapackage providing a grsecurity-patched Linux kernel for use
ii  securedrop-keyring                   0.1.1+0.7.0~rc1                            amd64        Provides an apt keyring for SecureDrop-related packages, so the master signing key used for SecureDrop packages can be updated via apt.
ii  securedrop-ossec-agent               2.8.2+0.7.0~rc1                            amd64        Installs the securedrop pre-configured OSSEC agent
ii  ssh-import-id                        3.21-0ubuntu1                              all          securely retrieve an SSH public key and install it locally
```

```
$ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
  securedrop-app-code
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.

```



Discussion:
It seems if I manually do a `sudo apt-get update` and then `sudo apt upgrade`, it pulls in the right dependencies and the upgraded package.
Did you figure out why your manual intervention was necessary @kushaldas? 
In my latest testing I could not reproduce this error. I am closing this issue.
Reopening because I also see this same behavior while upgrade testing `0.7.0~rc3` on hardware - the `securedrop-app-code` package was held back at 0.6. All other packages upgraded without intervention. But if so it's unclear why you didn't see this in further testing @kushaldas. Hypothesis: this is due to the addition of the `libjpeg-dev` dependency in 9880a30c8d58048ce33e02cdc940a26369a1d601
It looks like @emkll and @dachary also performed upgrade testing - did you experience this issue?
@redshiftzero yes, it is missing libjpeg-dev package dependency and that is why it is holding back. In my last testing `sudo cron-apt -s -i` actually did a `apt-get update`, so it found the proper dependencies and upgraded to the right `securedrop-app-code` package.
Do you understand why this only happens in some situations? `cron-apt -s -i` always does an `apt-get update` first, no? 
>Do you understand why this only happens in some situations? cron-apt -s -i always does an apt-get update first, no?

Yes (about doing that `apt-get update` by `cron-apt -s -i`), but, for some reason it failed for me once, I am guessing the same happened to you too.
> did you experience this issue?

I did not. Could it be that something installs libjpeg-dev when running ``vagrant up app-prod``?
yeah im able to also reproduce this on my instance... digging
Sooo I think this might be by design on debian's side ... I'm having trouble finding links explaining this behavior but as an analogy it seems to be similar to how permissions work on Android apps:

* You have an app `Foo` that you previously installed
* New updates to `Foo` come in without new permission changes and can be automatically updated
* Say one day `Foo` updates but is now requiring access to your `Camera` ... Android wants you to manually approve that change in underlying app dependency.

[This article](https://debian-administration.org/article/69/Some_upgrades_show_packages_being_kept_back) had a really good quote summing this up:

>  If I may try to rephrase: 'Apt-get dist-upgrade'ing also installs new packages brought in the chain of dependencies, whereas 'apt-get upgrade'ing only install newer versions of packages already installed. After a couple of years working with Debian, I never managed to exactly understand that difference. Would you believe I never found documentation who states this as simple?


Still diggin into solutions here
Hmm interesting - but then how did we add dependencies before? For example, when we added the [`securedrop-keyring`](https://github.com/freedomofpress/securedrop/commit/75961546e0078ed0c3ad47e8d67bb4a94c78bc2a#diff-8f4b8b485ba54d091bf0bd9e34f6b86b) package as a dependency of the `securedrop-app-code`?
To re-cap conversation that was conducted out of band.. the gist is this .. 

* In order to add any dependency to an existing installed package, a `dist-upgrade` needs to be run
* We do in-fact run a `dist-upgrade` as part of `cron-apt` but its very narrow in scope: `dist-upgrade -y -o APT::Get::Show-Upgraded=true -o Dir::Etc::SourceList=/etc/apt/security.list -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold`.
* That `security.list` file from above has a very select number of repos that are whitelisted:

```
root@app-staging:/home/vagrant# cat /etc/apt/security.list
deb http://security.ubuntu.com/ubuntu trusty-security main
deb-src http://security.ubuntu.com/ubuntu trusty-security main
deb http://security.ubuntu.com/ubuntu trusty-security universe
deb-src http://security.ubuntu.com/ubuntu trusty-security universe
deb [arch=amd64] https://apt.freedom.press trusty main
deb https://tor-apt.freedom.press trusty main
```

the problem is that the newly added dependency, `libjpeg-dev` is not in one of those repos, its under `archive.ubuntu.com/ubuntu trusty main` ... so apt-cron won't upgrade here.
* This worked previously for new depdendencies like `securedrop-keyring` because those were hosted under `apt.freedom.press` which is on the `security.list`.

Now theoretically we could add those new libjpeg-dev deps to our internal repo and things should work good -- but the consensus, this close to release, is to roll back the necessary changes that required this new dependency and punt solving this issue to a later release.

Resolving patch:
diff --git a/securedrop/journalist_app/admin.py b/securedrop/journalist_app/admin.py
--- a/securedrop/journalist_app/admin.py
+++ b/securedrop/journalist_app/admin.py
@@ -1,7 +1,5 @@
 # -*- coding: utf-8 -*-
 
-from PIL import Image
-
 import os
 
 from flask import (Blueprint, render_template, request, url_for, redirect, g,
@@ -33,19 +31,16 @@ def manage_config():
         form = LogoForm()
         if form.validate_on_submit():
             f = form.logo.data
-            custom_logo_filepath = os.path.join(config.SECUREDROP_ROOT,
-                                                "static/i/custom_logo.png")
+            custom_logo_filepath = os.path.join(current_app.static_folder, 'i',
+                                                'custom_logo.png')
             try:
-                with Image.open(f) as im:
-                    im.thumbnail((500, 450), resample=3)
-                    im.save(custom_logo_filepath, "PNG")
+                f.save(custom_logo_filepath)
                 flash(gettext("Image updated."), "logo-success")
             except Exception:
                 flash("Unable to process the image file."
                       " Try another one.", "logo-error")
             finally:
                 return redirect(url_for("admin.manage_config"))
-
         else:
             for field, errors in form.errors.items():
                 for error in errors:
diff --git a/securedrop/journalist_app/forms.py b/securedrop/journalist_app/forms.py
--- a/securedrop/journalist_app/forms.py
+++ b/securedrop/journalist_app/forms.py
@@ -58,7 +58,6 @@ class ReplyForm(FlaskForm):
 class LogoForm(FlaskForm):
     logo = FileField(validators=[
         FileRequired(message=gettext('File required.')),
-        FileAllowed(['jpg', 'png', 'jpeg'],
-                    message=gettext("You can only upload JPG/JPEG"
-                                    " or PNG image files."))
+        FileAllowed(['png'],
+                    message=gettext("Upload images only."))
     ])


Test patch:
diff --git a/securedrop/tests/test_journalist.py b/securedrop/tests/test_journalist.py
--- a/securedrop/tests/test_journalist.py
+++ b/securedrop/tests/test_journalist.py
@@ -1042,10 +1042,9 @@ def test_logo_upload_with_invalid_filetype_fails(self):
         resp = self.client.post(url_for('admin.manage_config'),
                                 data=form.data,
                                 follow_redirects=True)
-        self.assertMessageFlashed("You can only upload JPG/JPEG"
-                                  " or PNG image files.", "logo-error")
-        self.assertIn("You can only upload JPG/JPEG"
-                      " or PNG image files.", resp.data)
+        self.assertMessageFlashed("Upload images only.",
+                                  "logo-error")
+        self.assertIn("Upload images only.", resp.data)
 
     def test_logo_upload_with_empty_input_field_fails(self):
         self._login_admin()