giseldo/neodataset huggingface dataset:
SSE-C support (for AWS S3 backend) (lp:#1843343)

[Original report](https://bugs.launchpad.net/bugs/1843343) created by **Ivan Kurnosov (zerkms)**
```
AWS S3 SSE-C provides a better protection and I believe it would be
convenient to have its support together with SSE that is already available.

https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html

Thanks
```

Story points: 6.0
NameError: name 'unicode' is not defined (lp:#1838574)

[Original report](https://bugs.launchpad.net/bugs/1838574) created by **Stephan Müller (smueller18)**
```
Python 3 does not natively support the function unicode(). In
onedrivebacken.py this function is used which results in errors during
excecution. In Python 3, all occurences of unicode() must be replaced with
str().
In some projects I found the following code snippet which also may a
solution:

PY3 = sys.version_info[0] == 3

if PY3:
    unicode = str

Stacktrace:
Attempt 1 failed. NameError: name 'unicode' is not defined
Level 8:duplicity:Attempt 1 failed. NameError: name 'unicode' is not
defined
Traceback (most recent call last):
  File "/home/sm/git/lab/smueller18/docker-
duplicity/.venv/lib/python3.7/site-packages/duplicity/backend.py", line
375, in inner_retry
    return fn(self, *args)
  File "/home/sm/git/lab/smueller18/docker-
duplicity/.venv/lib/python3.7/site-packages/duplicity/backend.py", line
535, in put
    self.__do_put(source_path, remote_filename)
  File "/home/sm/git/lab/smueller18/docker-
duplicity/.venv/lib/python3.7/site-packages/duplicity/backend.py", line
521, in __do_put
    self.backend._put(source_path, remote_filename)
  File "/home/sm/git/lab/smueller18/docker-
duplicity/.venv/lib/python3.7/site-
packages/duplicity/backends/onedrivebackend.py", line 260, in _put
    u'Content-Length': unicode(len(chunk)),
NameError: name 'unicode' is not defined


Environment:
duplicity 0.8.02
Python 3.7.3
Ubuntu 19.04
```

Story points: 6.0
Permanent "Another duplicity instance is already running" (lp:#1837201)

[Original report](https://bugs.launchpad.net/bugs/1837201) created by **Eoghan Murray (eoghan-n)**
```
For many weeks now my duplicity instance (started via deja-dup) has been
failing with
"Another duplicity instance is already running with this archive directory"

My storage location is on a network which is automatically mounted

I checked the folder for the presence of a lockfile (after reading the
code) and there was none there.

It was only after adding extra info as follows:

`        log.FatalError(
            "Another duplicity instance is already running with this
archive directory: %s \n" % (globals.lockpath),
`

That I discovered that globals.lockpath is set to `~/.cache/deja-
dup/3ac8a04e2f4c8feb215d2e9f8eb12645/`

Should there be something in duplicity to test the age of the lockpath, or
actually test it for an actively running process, e.g. store the PID and
check (using ps) whether that process is running?

Apologies if this is a concern of deja-dup.
```

Story points: 14.0
typeerror: expected string or buffer from mega (lp:#1836785)

[Original report](https://bugs.launchpad.net/bugs/1836785) created by **David Parkin (davidmichaelparkin)**
```
Duplicity 0.4.14 Python 2.7.14
Commandline or args built around line 158 is a list. It appears that
megatools expects a string or buffer.

I used join to create a string and it appears to work. I notice that the
code hasn't changed so I assume no-one uses mega or I've something wrong.
```

Story points: 6.0
--dry-run does not work with restore (lp:#1836118)

[Original report](https://bugs.launchpad.net/bugs/1836118) created by **Jake Herrmann (jtherrmann-deactivatedaccount)**
```
The --dry-run option does not seem to work with the restore command. This
is the command I'm trying to run:

duplicity restore -vi --sign-key <sign key> --force <url> $HOME --dry-run

duplicity prints some info and then prompts for my passphrase, as usual.
After I enter my passphrase, the program exits without printing anything
else. I would expect it to instead print all of the files that would be
restored (because I've enabled info-level verbosity).

I'm running duplicity 0.7.11 on Debian GNU/Linux 9 (stretch).
```

Story points: 6.0
0.8.00 testsuite fails  with python 3.5.3 and gnupg 2.1.18 (lp:#1833998)

[Original report](https://bugs.launchpad.net/bugs/1833998) created by **az (az-debian)**
```
with python 3.5.3 and gnupg 2.1.18 the testsuite repeatably fails
unit/test_collections.py and almost all of unit/test_gpg.py and
unit/test_gpginterface.py.

the error messages all point to the same bit of code at
duplicity/gpginterface.py:447, so i'm just including the first one:

=================================== FAILURES
===================================
_____________________ CollectionTest.test_sigchain_fileobj
_____________________

self = <testing.unit.test_collections.CollectionTest
testMethod=test_sigchain_fileobj>

   @ pytest.mark.usefixtures(u"redirect_stdin")
    def test_sigchain_fileobj(self):
        u"""Test getting signature chain fileobjs from archive_dir_path"""
>       self.set_gpg_profile()

unit/test_collections.py:189:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _
unit/test_collections.py:97: in set_gpg_profile
    self.set_global(u'gpg_profile', gpg.GPGProfile(passphrase=u"foobar"))
../duplicity/gpg.py:95: in __init__
    self.gpg_version = self.get_gpg_version(globals.gpg_binary)
../duplicity/gpg.py:111: in get_gpg_version
    res = gnupg.run([u"--version"], create_fhs=[u"stdout"])
../duplicity/gpginterface.py:375: in run
    create_fhs, attach_fhs)
../duplicity/gpginterface.py:427: in _attach_fork_exec
    self._as_child(process, gnupg_commands, args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _

self = <duplicity.gpginterface.GnuPG object at 0x7fa25e8e52b0>
process = <duplicity.gpginterface.Process object at 0x7fa25f64cc88>
gnupg_commands = ['--version'], args = []

    def _as_child(self, process, gnupg_commands, args):
        u"""Stuff run after forking in child"""
        # child
        for std in _stds:
            p = process._pipes[std]
>           os.dup2(p.child, getattr(sys, u"__%s__" % std).fileno())
E           ValueError: underlying buffer has been detached

../duplicity/gpginterface.py:447: ValueError

with python 3.7.3 and gnupg 2.2.12 or 2.2.13 those same tests survive just
fine.
```

Story points: 6.0
[0.8] ParseArgsTest.test_unicode_paths_non_globbing is failing (lp:#1833562)

[Original report](https://bugs.launchpad.net/bugs/1833562) created by **Sebastien Bacher (seb128)**
```
Using 0.8 r1377

Trying on Ubuntu Disco to see how it behave the tests fail with a

$ cd /tmp
$ bzr branch lp:duplicity
$ cd duplicity
$ python2.7 setup.py build --force
$ PYTHONPATH=/tmp/duplicity/build/lib.linux-i686-2.7/ python2.7 ./setup.py
test
(that's on an i386 chroot, path to adapt for amd64)

The tests have some errors, including that one

self = <testing.unit.test_selection.ParseArgsTest
testMethod=test_unicode_paths_non_globbing>

    def test_unicode_paths_non_globbing(self):
        u"""Test functional test test_unicode_paths_non_globbing as a
unittest"""
        self.root = Path(u"testfiles/select-unicode")
        self.ParseTest([(u"--exclude", u"testfiles/select-
unicode/прыклад/пример/例/Παράδειγμα/उदाहरण.txt"),
                        (u"--exclude", u"testfiles/select-
unicode/прыклад/пример/例/Παράδειγμα/דוגמא.txt"),
                        (u"--exclude", u"testfiles/select-
unicode/прыклад/пример/例/მაგალითი/"),
                        (u"--include", u"testfiles/select-
unicode/прыклад/пример/例/"),
                        (u"--exclude", u"testfiles/select-
unicode/прыклад/пример/"),
                        (u"--include", u"testfiles/select-
unicode/прыклад/"),
                        (u"--include", u"testfiles/select-
unicode/օրինակ.txt"),
                        (u"--exclude", u"testfiles/select-unicode/**")],
                       [(), (u"прыклад",), (u"прыклад", u"пример"),
(u"прыклад", u"пример", u"例"),
                        (u"прыклад", u"пример", u"例", u"Παράδειγμα"),
                        (u"прыклад", u"пример", u"例", u"Παράδειγμα",
u"ઉદાહરણ.log"),
>                       (u"прыклад", u"উদাহরণ"), (u"օրինակ.txt",)])

unit/test_selection.py:887:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _
unit/test_selection.py:234: in ParseTest
    results_as_list = list(Iter.map(self.uc_index_from_path, self.Select))
../duplicity/lazy.py:49: in map
    yield function(i)
_ _ _ _ _ _ _ _ _ _ _ _
```

Story points: 6.0
TestSimpleUnicode.test_simple_unicode (lp:#1833561)

[Original report](https://bugs.launchpad.net/bugs/1833561) created by **Sebastien Bacher (seb128)**
```
Using 0.8 r1377

Trying on Ubuntu Disco to see how it behave the tests fail with a

$ cd /tmp
$ bzr branch lp:duplicity
$ cd duplicity
$ python2.7 setup.py build --force
$ PYTHONPATH=/tmp/duplicity/build/lib.linux-i686-2.7/ python2.7 ./setup.py
test
(that's on an i386 chroot, path to adapt for amd64)

The tests have some errors, including that one

____________________ TestSimpleUnicode.test_simple_unicode
_____________________

self = <testing.unit.test_globmatch.TestSimpleUnicode
testMethod=test_simple_unicode>

    def test_simple_unicode(self):
        u"""Test simple unicode comparison"""
self.assertEqual(inc_sel_file(u"прыклад/пример/例/Παράδειγμα/उदाहरण.txt",
>
u"прыклад/пример/例/Παράδειγμα/उदाहरण.txt"), 1)
E       AssertionError: None != 1

unit/test_globmatch.py:249: AssertionError


(TestSquareBrackets.test_square_bracket_options_unicode hits a similar
error)
```

Story points: 6.0
Incremental backups do not validate passphrase (lp:#1828761)

[Original report](https://bugs.launchpad.net/bugs/1828761) created by **Tom Boshoven (tomboshoven)**
```
This was reported in a comment on another bug, but I think this is serious
enough to warrant its own one. It broke my incremental backups and I just
happened to notice.
The report with repro steps is
https://bugs.launchpad.net/duplicity/+bug/918489/comments/22

The expected behavior would be to fail on invalid passphrases on
incremental backups (or otherwise make it very clear that this is the
user's responsibility).
I seem to remember testing this in the past and verifying that an error is
raised.
Right now, it leads to corruption and a beautiful stack trace followed by a
hanging application when trying to verify or restore:

GPG error detail: Traceback (innermost last):
  File "/bin/duplicity", line 1560, in <module>
    with_tempdir(main)
  File "/bin/duplicity", line 1546, in with_tempdir
    fn()
  File "/bin/duplicity", line 1398, in main
    do_backup(action)
  File "/bin/duplicity", line 1479, in do_backup
    verify(col_stats)
  File "/bin/duplicity", line 875, in verify
    for backup_ropath, current_path in collated:
  File "/usr/lib/python2.7/site-packages/duplicity/diffdir.py", line 276,
in collate2iters
    relem1 = riter1.next()
  File "/usr/lib/python2.7/site-packages/duplicity/patchdir.py", line 521,
in integrate_patch_iters
    for patch_seq in collated:
  File "/usr/lib/python2.7/site-packages/duplicity/diffdir.py", line 286,
in collate2iters
    relem2 = riter2.next()
  File "/usr/lib/python2.7/site-packages/duplicity/patchdir.py", line 121,
in difftar2path_iter
    tarinfo_list = [tar_iter.next()]
  File "/usr/lib/python2.7/site-packages/duplicity/patchdir.py", line 344,
in next
    self.set_tarfile()
  File "/usr/lib/python2.7/site-packages/duplicity/patchdir.py", line 332,
in set_tarfile
    assert not self.current_fp.close()
  File "/usr/lib/python2.7/site-packages/duplicity/dup_temp.py", line 227,
in close
    assert not self.fileobj.close()
  File "/usr/lib/python2.7/site-packages/duplicity/gpg.py", line 305, in
close
    self.gpg_failed()
  File "/usr/lib/python2.7/site-packages/duplicity/gpg.py", line 272, in
gpg_failed
    raise GPGError(msg)
 GPGError: GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: AES encrypted data
gpg: encrypted with 1 passphrase
gpg: decryption failed: Bad session key
===== End GnuPG log =====


GPGError: GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: AES encrypted data
gpg: encrypted with 1 passphrase
gpg: decryption failed: Bad session key
===== End GnuPG log =====


If left unnoticed, the only way to go about it is manual recovery.
I'm updating my backup scripts to do some basic validation of the
passphrase to make sure that a simple typo will not break my backups in the
future.

duplicity 0.7.18.2 on Arch Linux, Python 3.7.3.
```

Story points: 14.0
failure to remove backups and missing error info (remove-all-but-n-full) (lp:#1823560)

[Original report](https://bugs.launchpad.net/bugs/1823560) created by **alexander (bcclsn)**
```
duplicity remove-all-but-n-full 2 --force fails with an error that is not
specified...

Traceback (innermost last):
  File "/usr/bin/duplicity", line 1560, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1546, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1398, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1424, in do_backup
    action).set_values()
  File "/usr/lib/python2.7/site-packages/duplicity/collections.py", line
710, in set_values
    self.get_backup_chains(partials + backend_filename_list)
  File "/usr/lib/python2.7/site-packages/duplicity/collections.py", line
836, in get_backup_chains
    add_to_sets(f)
  File "/usr/lib/python2.7/site-packages/duplicity/collections.py", line
824, in add_to_sets
    if set.add_filename(filename):
  File "/usr/lib/python2.7/site-packages/duplicity/collections.py", line
105, in add_filename
    (self.volume_name_dict, filename)
 AssertionError: ({1: 'duplicity-full.20190407T165143Z.vol1.difftar.gpg',
2: 'duplicity-full.20190407T165143Z.vol2.difftar.gpg', 10: 'duplicity-
full.20190407T165143Z.vol10.difftar.gpg', 11: 'duplicity-
full.20190407T165143Z.vol11.difftar.gpg', 12: 'duplicity-
full.20190407T165143Z.vol12.difftar.gpg', 13: 'duplicity-
full.20190407T165143Z.vol13.difftar.gpg', 14: 'duplicity-
full.20190407T165143Z.vol14.difftar.gpg', 15: 'duplicity-
full.20190407T165143Z.vol15.difftar.gpg', 16: 'duplicity-
full.20190407T165143Z.vol16.difftar.gpg', 17: 'duplicity-
full.20190407T165143Z.vol17.difftar.gpg', 18: 'duplicity-
full.20190407T165143Z.vol18.difftar.gpg', 19: 'duplicity-
full.20190407T165143Z.vol19.difftar.gpg', 20: 'duplicity-
full.20190407T165143Z.vol20.difftar.gpg'}, 'duplicity-
full.20190407T165143Z.vol20.difftar.gpg')

duplicity version: 0.7.18.2
python version: 3.7.3
os distro: archlinux x64
```

Story points: 6.0
Failed with an unknown error. Traceback (innermost last): (lp:#1822077)

[Original report](https://bugs.launchpad.net/bugs/1822077) created by **Morgan Read (mstuff)**
```
Backup failed after running for over 12 hours...  So, it's going to start
all over again - again

(Please provide some way to manage auto initiated 'full backups' triggered
to guard against corruption - they're killing my data usage and storage.
See Bug #1821628)


Traceback (innermost last):
  File "/usr/bin/duplicity", line 1560, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1546, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1398, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1516, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 577, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 459, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py",
line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib64/python2.7/site-packages/duplicity/asyncscheduler.py",
line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 458, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 347, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 395,
in inner_retry
    % (n, e.__class__.__name__, util.uexc(e)))
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 82, in
uexc
    return ufn(m)
  File "/usr/lib64/python2.7/site-packages/duplicity/util.py", line 63, in
ufn
    return filename.decode(globals.fsencoding, 'replace')
 AttributeError: 'ProtocolError' object has no attribute 'decode'
```

Story points: 8.0
InvalidBackendURL error for B2 backend if application key contains a '/' character (lp:#1819390)

[Original report](https://bugs.launchpad.net/bugs/1819390) created by **Justin Warren (justin-eigenmagic)**
```
If your application key contains a '/' character, the parsing of the b2 URL
fails with this error:

InvalidBackendURL: Syntax error (port) in:
b2://<accountId>:<applicationKey>@<bucketname>

User error detail: Traceback (innermost last):
  File "/usr/bin/duplicity", line 1560, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1546, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    action = commandline.ProcessCommandLine(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line
1091, in ProcessCommandLine
    args = parse_cmdline_options(cmdline_list)
  File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line
720, in parse_cmdline_options
    lpath, backend_url = args_to_path_backend(args[0], args[1])  #@
UnusedVariable
  File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line
983, in args_to_path_backend
    arg1_is_backend, arg2_is_backend = backend.is_backend_url(arg1),
backend.is_backend_url(arg2)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 170,
in is_backend_url
    pu = ParsedUrl(url_string)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 306,
in __init__
    re.search('::[^:]+$', self.netloc), self.netloc))

Appears to be due to the regex in backend.py on line 306, but it's actually
due to the result returned by urlparse.urlparse(). netloc gets set to the
string of <accountid>: and the portion of the applicationKey up to (but not
including) the '/' character.

Essentially, '/' isn't a valid password character as far as urlparse() is
concerned.

Workaround is to delete the applicationKey in Backblaze and generate a new
one until you get an applicationKey that doesn't contain a '/' character.

Duplicity version: 0.7.18.2, also appears in 0.7.17 from Ubuntu bionic
Python version: 2.7.15rc1
OS: Ubuntu 18.04.2 LTS (bionic)
Target filesystem: Backblaze
```

Story points: 16.0
duplicity does not handle accentuated character for folder path to save (lp:#1816232)

[Original report](https://bugs.launchpad.net/bugs/1816232) created by **Thierry B. (thierrybo2)**
```
Duplicity version 0.8-series. (rev 1348)
Pyhon 2.7
Distro: Devuan GNU/Linux ascii

duplicity --verbosity debug --num-retries 3 --asynchronous-upload --cf-
backend pca --volsize 100 ~/Téléchargements/ swift://thierrybo-desk_tests
/usr/local/bin/duplicity:69: UnicodeWarning: Unicode equal comparison
failed to convert both arguments to Unicode - interpreting them as being
unequal
  if u'--pydevd' in sys.argv or os.getenv(u'PYDEVD', None):
Using temporary directory /tmp/duplicity-_0Apzy-tempdir
Traceback (innermost last):
  File "/usr/local/bin/duplicity", line 1678, in <module>
    with_tempdir(main)
  File "/usr/local/bin/duplicity", line 1664, in with_tempdir
    fn()
  File "/usr/local/bin/duplicity", line 1497, in main
    action = commandline.ProcessCommandLine(sys.argv[1:])
  File "/usr/local/lib/python2.7/dist-packages/duplicity/commandline.py",
line 1146, in ProcessCommandLine
    args = parse_cmdline_options(cmdline_list)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/commandline.py",
line 698, in parse_cmdline_options
    possible = [c for c in commands if c.startswith(cmd)]
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 17:
ordinal not in range(128)


If I replace "~/Téléchargements/" with a path without accented  characters
it works
```

Story points: 18.0
Stream archive to backend while it is being created (lp:#1814792)

[Original report](https://bugs.launchpad.net/bugs/1814792) created by **Giovanni Mascellani (giomasce)**
```
It currently seems that archives generated by duplicity are first encrypted
with GPG and stored in the temporary file, then read again and sent to the
backend.

I wonder if it is possible, at least for backends that support such
operation, to directly stream the archive to the backend while it is being
created and encrypted, without having to wait for the full encryption to
finish.

This would have several advantages, it seems: first, there would be less
total IO, because there would be no need for temporary files; second, IO
and CPU usage would be levelled by network connection (which is supposedly
slower), instead of spiking while the disk is read and gpg is executed;
third, network (again supposed to be the bottleneck factor) would be better
used, because it would not have to wait for the archive generation and
encryption. More in general, the total running time would be the maximum
between the archive generation and uploading time, instead of their sum,
and, as I said, resources would be more levelled.

Thanks for duplicity, it is really a nice tool!
```
Original tags: wishlist

Story points: 6.0
Feature Request: More Informative Notice Level (lp:#1811643)

[Original report](https://bugs.launchpad.net/bugs/1811643) created by **Fedaykin (fedaykin7c2)**
```
It would be much more practical if Notice Level (4) would contain
additional pieces of information about backup in progress like:

Args
collection-status
Processed volume
and also upload speed for volume.
```
Original tags: feature

Story points: 6.0
Rename manifest file (lp:#1810511)

[Original report](https://bugs.launchpad.net/bugs/1810511) created by **Anthony (0othan)**
```
Hello all,

Is this possible to rename manifest file?

Indeed, the current manifest naming is duplicity-
inc.XXXXZ.to.YYYYZ.manifest.

The issue with S3 is that we cannot ignore manifest when we move to glacier
due to convention naming..

Indeed, we can add a filter by object (by example for signature:
PATH/duplicity-new-signatures | PATH/duplicity-full-signatures) but regex
are not used: we cannot use PATH/*.manifest to filter manifest file so if
we use PATH/duplicity-full, manifest & volumes will be excluded during move
to glacier..

The idea is to ignore manifest/signature file from glacier policy to avoid
to wait 2/3h in the case that we need to check backups on other desktop by
example... For now, we can add a filter only for signature...

Best regards,
```

Story points: 6.0