giseldo/neodataset huggingface dataset:
KeyError while running verify (lp:#1632858)

[Original report](https://bugs.launchpad.net/bugs/1632858) created by **Pastafarianist (pastafarianist)**
```
WebDAV GET /path/to/backup.vol579.difftar.gpg request with headers:
{'Connection': 'keep-alive', 'Authorization': 'Basic <long-string-here>'}
WebDAV data length: 4
WebDAV response status 200 with reason 'OK'.
Deleting /tmp/duplicity-7CltV8-tempdir/mktemp-M2iDvp-592
Processed volume 585 of 4543
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1530, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1524, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1378, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1457, in do_backup
    verify(col_stats)
  File "/usr/bin/duplicity", line 860, in verify
    for backup_ropath, current_path in collated:
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 276,
in collate2iters
    relem1 = riter1.next()
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 516,
in integrate_patch_iters
    for patch_seq in collated:
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 388,
in yield_tuples
    setrorps(overflow, elems)
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 377,
in setrorps
    elems[i] = iter_list[i].next()
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 127,
in difftar2path_iter
    multivol_fileobj.close()  # aborting in middle of multivol
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 248,
in close
    if not self.addtobuffer():
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 236,
in addtobuffer
    self.tarinfo_list[0] = self.tar_iter.next()
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 343,
in next
    self.set_tarfile()
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 332,
in set_tarfile
    self.current_fp = self.fileobj_iter.next()
  File "/usr/bin/duplicity", line 759, in get_fileobj_iter
    backup_set.volume_name_dict[vol_num],
KeyError: 580

duplicity 0.7.03
Python 2.7.9
Debian Jessie
Target filesystem: WebDAV
```

Story points: 20.0
Backup on Backblaze B2 don't work on CentoS 6.8  (error : UnsupportedBackendScheme) (lp:#1632475)

[Original report](https://bugs.launchpad.net/bugs/1632475) created by **Gilbert Reims (gilbert721)**
```
Hi,
As I see on internet, if I want to use the last version of Duplicity with a
VPS server on CentOS 6.8 (or 6.x) I need to install it manually with all
dependencies.

I follow this tutorial :
http://blog.helge.net/2014/02/how-to-install-duplicity-on-centos.html
or
http://clients.microtronix-
tech.com/knowledgebase.php?action=displayarticle&id=5

It works and Duplicity works fine with FTP backup, and seems to work with
all functions.

If I try to use Duplicity with backblaze 2, I have this error message when
I launch a .sh script :
UnsupportedBackendScheme: scheme not supported in url:
b2://xxxxx:xxxxx@xxxx/xxxx/

But on another VPS with CentOS Linux release 7.2.1511, it works with B2.

On the 2 servers I have Duplicity 0.7.10.

Do you have an idea of the problem ? Maybe I need to add another
dependencies ?
Thank you in advance for your answers,
Regards, Gilbert
Sorry if i need to ask question instead report a bug !
```

Story points: 6.0
Cross-device link error in LocalBackend (lp:#1631472)

[Original report](https://bugs.launchpad.net/bugs/1631472) created by **Igor (ertong)**
```
duplicity 0.7.10 (August 20, 2016)
Linux backup 3.13.0-46-generic #79-Ubuntu SMP Tue Mar 10 20:06:50 UTC 2015
x86_64 x86_64
/usr/bin/python2 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2]


When we use duplicity with LocalBackend and backup folder is mounted to
different drive than archivedir, duplicity fails to copy signatures to
backup folder.

In order to find the problem,
1. I've added some messages in localbackend.py: http://take.ms/QMCMU
2. Run duplicity with verbose output and LocalBackend file:///... pointing
to directory on dedicated hdd
3. I receive output like http://take.ms/92JW5 , where we clearly see
"[Errno 18] Invalid cross-device link" error message
4. My backup have no signature files


If I explicitly set --archive-dir and --tempdir to the same hard drive as
backup directory, everything works fine.

If I use different backend, everything works fine.
```

Story points: 6.0
Restarts fail because comparing long key ID to short key ID (lp:#1631414)

[Original report](https://bugs.launchpad.net/bugs/1631414) created by **Dan Watkins (oddbloke)**
```
Using duplicity 0.7.10 on Ubuntu yakkety, when a restart is attempted, I
see a failure because the 8 character key ID is being compared against the
16 character key ID.  See the relevant part of my log below:

--- Start running command BKP at 16:24:00.197 ---
Reading globbing filelist <snip>/exclude
Synchronising remote metadata to local cache...
Copying duplicity-inc.20161006T075558Z.to.20161007T091252Z.manifest.gpg to
local cache.
Copying duplicity-new-
signatures.20161005T135758Z.to.20161006T075558Z.sigtar.gpg to local cache.
Copying duplicity-new-
signatures.20161006T075558Z.to.20161007T091252Z.sigtar.gpg to local cache.
Last full backup left a partial set, restarting.
Last full backup date: Fri Oct  7 15:19:21 2016
Reuse configured PASSPHRASE as SIGN_PASSPHRASE
RESTART: Volumes 3 to 3 failed to upload before termination.
         Restarting backup at volume 3.
Volume was signed by key 52DA9A50, not 4F07B22452DA9A50
16:25:53.466 Task 'BKP' failed with exit code '22'.
--- Finished state FAILED 'code 22' at 16:25:53.466 - Runtime 00:01:53.269
---
```

Story points: 20.0
Cloud authentication isn't renegotiated on long backups (lp:#1630002)

[Original report](https://bugs.launchpad.net/bugs/1630002) created by **Simon Greenwood (sfgreenwood-gmail)**
```
Version 0.7.10
Python 2.7.9
Debian 8.6 (jessie)

Tested backends: Backblaze, Hubic

I have a couple of large backups of around 300GB and 500GB that I am trying
to create on cloud storage. I have tested with Backblaze B2 and Hubic as
both seem to offer good value for money and as they work with duplicity.
Both have token based authentication that times out after a period - on
Backblaze it seems to be 12 hours and on Hubic it seems to be 48. Neither
backend renegotiates once it starts failing. I would imagine that this
applies to other cloud services.

A common issue is that negotiation occurs in the init function and isn't
tried again once the cloud service rejects the connection when the
authentication period times out. A fix for this should be that the
negotiation function is separated out and then can be called when the
service rejects the connection, usually with a 403 error. I have started
work on this for Backblaze.

However, it also seems, certainly with the Backblaze backend, that resuming
a failed backup doesn't initialise the connection and that the
authentication token isn't renegotiated so it may also be the same with
other services.
```

Story points: 6.0
Please support running backup as root but dropping privilege for gpg/network/etc (lp:#1629151)

[Original report](https://bugs.launchpad.net/bugs/1629151) created by **Josh Triplett (joshtriplett)**
```
In order to back up files owned by multiple users, I need to start
duplicity as root.  However, I don't want to run gpg, ssh, or a network
connection to a storage service as root.

Please consider offering a duplicity option to use root privileges to read
the data to back up (or to write out restored data with restored ownership
and permissions), but to drop privileges for all other operations (gpg,
ssh, network connections, anything other than reading and writing the data
to backup/restore).
```

Story points: 8.0
No incremental when backing up to Hubic (lp:#1624865)

[Original report](https://bugs.launchpad.net/bugs/1624865) created by **David (davfrombe)**
```
Hi,

Using duplicity on Ubuntu 16.04, I can successfully backup on Hubic but
only full backups are created. Every successive run creates a new full
backup.

duplicity collection-status returns "Last full backup date: none"

Ho do I troubleshoot this so I can give you guys more details/log ?

Duplicity version : duplicity 0.7.01
Python version : Python 2.7.9
OS Distro and version : Ubuntu server 16.04 LTS
Type of target filesystem: Hubic cloud storage
Command line : duplicity -v9 ~/test.txt cf+hubic://backup/

Regards,

David
```

Story points: 8.0
Corrupted local cached file causes AssertionError (lp:#1622133)

[Original report](https://bugs.launchpad.net/bugs/1622133) created by **Phill (phill.l)**
```
After the following error (shown here with just a status command but
happens with backups too), duplicity never recovers and reports the same
error every time it is run:

$ duplicity collection-status ***REDACTED***
Local and Remote metadata are synchronized, no sync needed.
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1532, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1526, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1380, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1405, in do_backup
    globals.archive_dir).set_values()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line
710, in set_values
    self.get_backup_chains(partials + backend_filename_list)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line
835, in get_backup_chains
    add_to_sets(f)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line
823, in add_to_sets
    if set.add_filename(filename):
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line
101, in add_filename
    self.set_manifest(filename)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line
136, in set_manifest
    remote_filename)
AssertionError: ('duplicity-
inc.20160910T030003Z.to.20160910T040004Z.manifest.part', 'duplicity-
inc.20160910T030003Z.to.20160910T040004Z.manifest.gpg')

I deleted the ".part" file from the local cache and duplicity worked from
there after. The ".gpg" file did not exist.

I'm reporting this as a bug because I think it is reasonable to expect
programs to manage their own cache remove any cached content when it
detects a problem.

Unfortunately, I didn't take a copy of the offending file so I'm unable to
assist in what the corruption was. However, after deleting the file and
running the backup, no such file exists in the cache.

$ duplicity --version
duplicity 0.7.06

$ sudo apt list | grep duplicity
duplicity/xenial,now 0.7.06-2ubuntu2 amd64 [installed]

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"
```

Story points: 6.0
gio backend: please don't run dbus-launch unconditionally (lp:#1622074)

[Original report](https://bugs.launchpad.net/bugs/1622074) created by **az (az-debian)**
```
this is a forward of debian bug #836088, which lives over there:
http://bugs.debian.org/836088
the original reporter asks for improvements in the gio/dbus interaction
logic, namely this:

---
As described in <https://lists.debian.org/debian-
devel/2016/08/msg00554.html>
I'm trying to reduce how much dbus-launch is used in Debian.
duplicity's gio backend currently runs dbus-launch if
DBUS_SESSION_BUS_ADDRESS is unset.

This is only a minor bug in Debian, because Debian's dbus packaging is
designed to ensure that either DBUS_SESSION_BUS_ADDRESS is set for normal
sessions, or dbus-launch isn't available anyway. However, it could cause
problems in edge cases and is probably worth fixing upstream.

One issue with the current code is that it second-guesses how dbus
itself will find the session bus. In recent libdbus and GDBus, the
fallback behaviour if DBUS_SESSION_BUS_ADDRESS is unset is to look
for $XDG_RUNTIME_DIR/bus: if the environment variable is set, that
directory contains ./bus, and ./bus is a socket owned by the current
uid, then libdbus and GDBus will automatically use it. In particular,
the dbus-user-session Debian package sets up this situation. duplicity
should not run dbus-launch without first looking for that socket.

Another issue with the approach duplicity has taken is that it relies
on dbus-launch, which is X11-specific legacy code that does several
things, none of them particularly well.
This Flatpak commit illustrates how eval `dbus-launch` can be replaced
by invoking dbus-daemon directly, avoiding the X11-specific dbus-launch:
<https://github.com/flatpak/flatpak/commit/6cc8062cfb3f9410d54a27e7ccca77c103e441e8>.
It is possible to send both the address and the pid to stdout
("--print-address=1 --print-pid=1") if that would be easier to do from
Python; their order is undocumented but predictable, and I don't intend
to break it in future D-Bus releases.

Alternatively, and perhaps most simply, duplicity could stop trying to
compensate for a missing session bus address at all. On systems with
$XDG_RUNTIME_DIR/bus, it would "just work" anyway; or when run under X11
(even with no dbus-daemon running), X11 autolaunching would create a
dbus-daemon anyway; or if duplicity is being run in a non-GUI
environment, for example from cron, its documentation could mention that
use of the gio backend sometimes requires using

    dbus-run-session -- duplicity [ARGS...]

which has been available since dbus 1.8, and automatically cleans up
the dbus-daemon after duplicity terminates (successfully or not).
---
```

Story points: 6.0
SIGINT sometimes ignored when writing to AWS S3 backends (lp:#1618951)

[Original report](https://bugs.launchpad.net/bugs/1618951) created by **Saj Goonatilleke (saj-r)**
```
While not documented in the duplicity(1) manual, signalling duplicity
with SIGINT _should_ initiate a graceful shutdown:

https://lists.nongnu.org/archive/html/duplicity-talk/2014-10/msg00014.html

http://bazaar.launchpad.net/~duplicity-
team/duplicity/0.7-series/view/1240/bin/duplicity#L1558

Unfortunately, due in part to buggy third-party libraries, this does not
always work.

Steps to reproduce:

  1. Configure duplicity to back up to an AWS S3 backend.

  2. Using whatever method you prefer, set up a filter to drop network
     traffic from duplicity to its backend.  (So as to simulate a
     partial network failure.)

  3. Invoke 'duplicity collection-status' against the AWS S3 backend.
     duplicity should stall here.

  4. Signal the duplicity process with SIGINT.

Expected behaviour:

  duplicity removes ephemeral state and locks from the local filesystem
  then self-terminates with non-zero exit status.

Observed behaviour:

  duplicity fails to terminate.

Environment:

  - duplicity 0.7.09
  - boto 2.41.0
  - Python 2.7.5
  - Linux (EL7)

Analysis:

  duplicity expects KeyboardInterrupt to propagate up the stack once the
  Python runtime is signalled with SIGINT.  If, at the point the SIGINT
  is handled by the Python runtime, duplicity is busy executing boto
  library routines, the KeyboardInterrupt may be gobbled by a greedy
  'except' clause:

https://github.com/boto/boto/blob/abb38474ee5124bb571da0c42be67cd27c47094f/boto/s3/connection.py#L574-L577

  In this case, duplicity will never see the KeyboardInterrupt.

  A boto bug was filed for the catch-all in May of 2014:

  https://github.com/boto/boto/issues/2262

  (This is a particularly egregous catch-all.  'except Exception:' would
  have been enough to avoid this particular problem on CPython >= 2.5.)

I have not scanned other third-party imports to check whether this
problem affects more backend types.

As a workaround for this problem, our process supervisor was modified to
signal duplicity with SIGTERM then perform its own cleanup as best as it
could.  This arrangement is fragile:  what we assume of the current
duplicity release may not hold in the future (e.g.: lock file paths).

In an effort to guarantee behaviour, would it be possible to install a
custom handler for SIGINT/SIGTERM?  This handler would be immune to
third-party blunders, though I imagine it would complicate the code.
```

Story points: 6.0
OneDrive: Often asks for reauth (lp:#1616664)

[Original report](https://bugs.launchpad.net/bugs/1616664) created by **Sven (n-ubuntu-one)**
```
First of all thanks for the great software and thanks for the OneDrive
plugin. OneDrive support is quite rare (especially if one wants to access
it using a Linux server).

root@home ~/backups # duplicity --version
duplicity 0.7.09
root@home ~/backups # uname -a
Linux home 3.16.0-0.bpo.4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u3~bpo70+1
(2016-07-07) x86_64 GNU/Linux
root@home ~/backups # lsb_release -a
Distributor ID: Debian
Description:    Debian GNU/Linux 7.11 (wheezy)
Release:        7.11
Codename:       wheezy
root@home ~/backups # python --version
Python 2.7.3

I get this message from time to time (a few times per day):

> In order to authorize duplicity to access your OneDrive, please open
[url] in a browser and copy the URL of the blank page the dialog leads to.

The thing is every time I can (/need to) just hit enter without inputting
anything and the upload continues. Even after quitting and letting
duplicity continue by restarting it the upload just works fine again
without interaction.

The main problem here is that one has to check on a regular basis whether
such a request blocks the upload progress (In my case the upload is slow
and the whole process takes days, sometimes upto weeks and this costs me a
whole day without any progress if I don't have a look at it often enough).

It would be very helpful if this could get fixed.

Another message that's comes up often (much more often) is this one:

> Attempt 1 failed. Error: []

It doesn't has negative consequences and doesn't require manual
interaction, but I'd at least expect it to show an error code instead of an
empty list/array/object.
```
Original tags: onedrive

Story points: 6.0
Sauvegarde (lp:#1614607)

[Original report](https://bugs.launchpad.net/bugs/1614607) created by **Gava Daniel (gava-daniel)**
```
Bonjour,

Deja Dup ,fonctionnait très bien jusqu'à ce matin.Plusieurs tentatives
n'ont fait que confirmé l'echec pour une raison inconnue.Je vous fait
suivre le rapport de Deja Dup.Je suis sous Ubuntu 16.04 depuis
peu.file:///home/gava/Images/Capture%20du%202016-08-18%2015-29-25.png
/home/gava/Images/Capture du 2016-08-18 15-29-25.png
```

Story points: 6.0
Incremental backup prevents "restore-missing" from restoring files, that were deleted before (lp:#1610667)

[Original report](https://bugs.launchpad.net/bugs/1610667) created by **Vej (vej)**
```
If I do the following, I get an empty list.

1. Perform a fresh backup to a new folder (set a password for encryption).
2. Create the folder test and the file test/testfile.txt.
3. Restart the Backup (perform an incremental Backup)
4. Delete the file.
5. Restart the Backup (perform an incremental Backup)
6. Try to restore the file using LC_ALL=C deja-dup --restore-missing test/

Result: The list is empty.

I expect to see the file testfile.txt in the list, so I can restore it
using the backup created in Step 3.

Some other users of the German community verified this bug (see
https://forum.ubuntuusers.de/topic/fragen-zu-d-j-dup/ for an example - in
German).

I already mentioned this in bug #1377873, but that one is marked as "Fix
Released", so ML suggested to open a new one.


Ubuntu 16.04.1 LTS
deja-dup    34.2-0ubuntu1
duplicity   0.7.06-2ubuntu2

gsettings list-recursively org.gnome.DejaDup:

org.gnome.DejaDup last-restore '2015-11-01T21:37:49.910024Z'
org.gnome.DejaDup periodic true
org.gnome.DejaDup full-backup-period 75
org.gnome.DejaDup backend 'file'
org.gnome.DejaDup last-run '2016-08-07T10:21:31.709004Z'
org.gnome.DejaDup nag-check '2016-07-31T21:55:33.643340Z'
org.gnome.DejaDup prompt-check '2013-01-27T20:24:01.762209Z'
org.gnome.DejaDup root-prompt true
org.gnome.DejaDup include-list <A list including my home, /etc and a few
additional locations>
org.gnome.DejaDup exclude-list <A list excluding Trash and a few
directories>
org.gnome.DejaDup last-backup '2016-08-07T10:21:31.709004Z'
org.gnome.DejaDup periodic-period 1
org.gnome.DejaDup delete-after 730
[...]
org.gnome.DejaDup.File path 'file:///media/user/extern/Deja-Backup'
org.gnome.DejaDup.File short-name 'extern'
org.gnome.DejaDup.File uuid <a valid UUID>
org.gnome.DejaDup.File icon '. GThemedIcon drive-harddisk-usb drive-
harddisk drive'
org.gnome.DejaDup.File relpath b'Desktop/extern'
org.gnome.DejaDup.File name 'Vendor External USB 3.0: extern'
org.gnome.DejaDup.File type 'volume'
```
Original tags: restore

Story points: 22.0
b2 backend: dies with unicode error (lp:#1605985)

[Original report](https://bugs.launchpad.net/bugs/1605985) created by **az (az-debian)**
```
this is a forward of debian bug #830896, which lives over there:
http://bugs.debian.org/830896

the original reporter "created a Backblaze B2 Cloud Storage account and
tried to use it with
duplicity.  However all duplicity commands using B2 fail for me with the
same
error messages:

/usr/lib/python2.7/dist-packages/duplicity/backends/b2backend.py:211:
UnicodeWarning: Unicode equal comparison failed to convert both arguments
to Unicode - interpreting them as being unequal
  if bucket_name not in bucket_names:
FatalBackendException: Bucket cannot be created"
```

Story points: 6.0
Performance issue while handling big storages. (lp:#1605133)

[Original report](https://bugs.launchpad.net/bugs/1605133) created by **Grzegorz Żyszkiewicz (grzegorz-zyszkiewicz)**
```
There is a performance issue while handling big collection.
By big I mean 10000+ files(3000 different full backups) already backed up
by duplicity.

This issue is caused by unnecessarily parsing filename in
collections.BackupSet.add_filename.
In function CollectionsStatus.get_backup_chains a set of BackupSet is
created and each file is verified if fits any of already existing BackupSet
by calling method collections.BackupSet.add_filename for each filename in
list. This cause unnecessarily filename parsing for same file up to N times
(where N is number of full backups).

It can be avoided by parsing all filenames beforehand and passing already
parsed filename along with raw filename into
CollectionsStatus.get_backup_chains.add_to_sets and further to
collections.BackupSet.add_filename.

Same mechanism already exists within SignatureChain.add_filename.

It decreased duplicity working time from 22 minutes to 8 minutes in my
case.
```

Story points: 6.0
Full backup missing config files and hidden directories (lp:#1602111)

[Original report](https://bugs.launchpad.net/bugs/1602111) created by **Anes Lihovac (anes-lihovac-gmail)**
```
I have selected that my home directory gets backuped daily only excluding
~/Downloads. After an erase of my home and restoring everything, I am
missing everything under ~/.local/share/*

System information:

duplicity 0.7.02
Python 2.7.10
Ubuntu 15.10 + encrypted home directory, the backup is a separate disc.
```

Story points: 6.0