giseldo/neodataset huggingface dataset:
Replace optparse with argparse for python 2.7+ (lp:#1480565)

[Original report](https://bugs.launchpad.net/bugs/1480565) created by **Aaron Whitehouse (aaron-whitehouse)**
```
As of python 2.7, optparse is deprecated in favour of argparse:
https://docs.python.org/2/library/optparse.html
https://docs.python.org/2/library/argparse.html

At some point, we should therefore change commandline.py to use argparse
instead of optparse.

Note that argparse is new in 2.7, so isn't included by default in python
2.6 and below. It is, however, available as a separate package that is
compatible with python 2.3+. Options for supporting python 2.6 after
updating commandline.py would be to require the installation of the
separate package, or include a copy of the module with duplicity.
```

Story points: 6.0
collection-status removes local running backup files (sigtar) (lp:#1471818)

[Original report](https://bugs.launchpad.net/bugs/1471818) created by **John Jasen (jjasen)**
```
Duplicity: 0.6.18-3 (debian stock)
Python: 2.7.3-4+deb7 (debian stock)
OS: Debian GNU/Linux 7.8 (wheezy)

We run duplicity --collection-status as part of periodic system health cron
job (to feed passive nagios checks). The health script runs every 5
minutes.

During the normal backup run, on the local disk, signatures are written to
$filename.sigtar.part. That file is then copied to $filename.sigtar and
gzipped.

If $filename.sigtar.part is being copied to $filename.sigtar during a
collection-status, duplicity will remove $filename.sigtar.  These cause
local backups to show as failed/incomplete, and for duplicity to fail with
an error -- usually something along the lines of:

"OSError: [Errno 2] No such file or directory:
'/var/cache/duplicity/$hostname/@/duplicity-full-signatures.$date-time-
stamp.sigtar'"

This can be repeated, perhaps, by unzipping $filename.sigtar.gz and running
collection-status. It can definitely be repeated by cp
$filename.sigtar.part to $filename.sigtar, and running collection-status.
```

Story points: 6.0
TotalDestinationSizeChange doesn't count par2 data (lp:#1465165)

[Original report](https://bugs.launchpad.net/bugs/1465165) created by **Kuang-che Wu (kcwu)**
```
Currently, TotalDestinationSizeChange is just summed up the data volume.
However, backends may have additional overhead specific to each backend.
Especially, the par2 redudency data files generated by par2 backend are not
counted in TotalDestinationSizeChange.

In other words, the general form of this problem is,
TotalDestinationSizeChange should be calculated by backends.
```

Story points: 6.0
pydrive backend uploads volumes multiple times (lp:#1462862)

[Original report](https://bugs.launchpad.net/bugs/1462862) created by **Zhongfu Li (zhongfu)**
```
When duplicity is backing up with pydrive, it occasionally uploads volumes
and updates the manifest with those volumes multiple times (although I've
only seen it done twice, and usually one volume at a time). I've not
exactly pinpointed the cause, but it seems like something to do with
asynchronous uploads (even though I have not explicitly passed
--asynchronous-upload).

It seems to happen randomly, but one way that I've managed to reproduce
this is to stop the backup and restart it later.

When this happens, I see duplicates of the volumes in question, both on
Google Drive (as Google Drive allows multiple files in the same folder with
the same name) and in the manifest (where the entries are repeated, but
with differing checksums). Sometimes, one of the tar archives are
incomplete (reports "a lone zero block"), but when it's induced by
restarting the backup, the volumes seem to extract without any errors. A
file with an incorrect checksum might also be uploaded, but this is very
rare in my (short) experience.

--------------------------------------------------
Duplicity version: duplicity 0.7.03+bzr1099 (latest version from trunk ppa)
Python version: Python 2.7.9
OS: Ubuntu 15.04 vivid
Target filesystem: Linux, btrfs (to Google Drive)

(Apologies for only providing a log for verbosity 5 -- it's quite
troublesome to attempt to reproduce this, but I can do it at your request)
```
Original tags: duply gdocs pydrive

Story points: 10.0
webdav fails updating hidden directories (lp:#1460489)

[Original report](https://bugs.launchpad.net/bugs/1460489) created by **JP (0j-p)**
```
Duplicity version:  duplicity 0.7.02
Python version:  Python 2.7.9
OS Distro and version:  Ubuntu 15.04
Type of target filesystem: webdav

duplicity -v9 "/home/foo/.session/"
"webdavs://user:pass@site.url/foo/bar/.session"
WebDAV response status 409 with reason 'Conflict'.
Backtrace of previous error: Traceback (innermost last):
  File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py", line
365, in inner_retry
    return fn(self, *args)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py", line
515, in put
    self.__do_put(source_path, remote_filename)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py", line
501, in __do_put
    self.backend._put(source_path, remote_filename)
  File "/usr/local/lib/python2.7/dist-
packages/duplicity/backends/webdavbackend.py", line 409, in _put
    raise e
 BackendException: Bad status code 409 reason Conflict.

Attempt 1 failed. BackendException: Bad status code 409 reason Conflict.
```
Original tags: duplicity webdav

Story points: 6.0
Backend should use subfolders when available (lp:#1458507)

[Original report](https://bugs.launchpad.net/bugs/1458507) created by **Diego Peinador (diegopeinador)**
```
Some backends can't handle many files in a single folder (i.e. FTP) and
when this limit is reached Duplicity can't check existing backups and all
new backups are full (even when Duplicity is configured to create
incremental backups if last full isn't so old).

This limitation in the backend can be bypassed if duplicity could use
subfolders to store its data. For instance one folder with the manifest
files and then volumes for each backup in their own folders.

In some ftp backends this couldn't be enough, and a bigger default volume
size may come in handy.
```

Story points: 6.0
Deja-dup fails backing up (lp:#1457185)

[Original report](https://bugs.launchpad.net/bugs/1457185) created by **Wolf Rogner (war-rsb)**
```
Deja-dup fails with:

Failed to read /tmp/duplicity-dVcqOt-tempdir/mktemp-DUF_rW-325: (<type
'exceptions.IOError'>, IOError('CRC check failed 0xd1a0d155 !=
0x4c0714ccL',), <traceback object at 0x7f2240758098>)

This is an error already reported in 2010 somewhere else.

tried to mitigate by reinstalling, clearing caches and parameters to no
avail

ProblemType: Bug
DistroRelease: Ubuntu 15.04
Package: deja-dup 32.0-0ubuntu5
ProcVersionSignature: Ubuntu 3.19.0-18.18-generic 3.19.6
Uname: Linux 3.19.0-18-generic x86_64
NonfreeKernelModules: wl
ApportVersion: 2.17.2-0ubuntu1
Architecture: amd64
CurrentDesktop: Unity
Date: Wed May 20 20:58:37 2015
InstallationDate: Installed on 2013-05-28 (721 days ago)
InstallationMedia: Ubuntu 13.04 "Raring Ringtail" - Release amd64+mac
(20130424)
SourcePackage: deja-dup
UpgradeStatus: Upgraded to vivid on 2015-04-25 (25 days ago)
```
Original tags: amd64 apport-bug vivid

Story points: 6.0
Backup stops in the middle with a python traceback (lp:#1453114)

[Original report](https://bugs.launchpad.net/bugs/1453114) created by **Shoham Peller (shoham-peller)**
```
Backup Failed:
Failed with an unknown error.

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1494, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1488, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1337, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1458, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 542, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 403, in write_multivol
    globals.gpg_profile, globals.volsize)
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 327, in
GPGWriteFile
    bytes_to_go = data_size - get_current_size()
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 320, in
get_current_size
    return os.stat(filename).st_size
OSError: [Errno 2] No such file or directory: '/tmp/duplicity-x6bEln-
tempdir/mktemp-3XerZ6-13'

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: deja-dup 30.0-0ubuntu4
ProcVersionSignature: Ubuntu 3.13.0-39.66-generic 3.13.11.8
Uname: Linux 3.13.0-39-generic x86_64
NonfreeKernelModules: nvidia
ApportVersion: 2.14.1-0ubuntu3.10
Architecture: amd64
Date: Fri May  8 14:52:01 2015
InstallationDate: Installed on 2014-11-05 (184 days ago)
InstallationMedia: Ubuntu 14.04.1 LTS "Trusty Tahr" - Release amd64
(20140722.2)
SourcePackage: deja-dup
UpgradeStatus: No upgrade log present (probably fresh install)
```
Original tags: amd64 apport-bug trusty

Story points: 6.0
Unknown Error during backup (lp:#1450702)

[Original report](https://bugs.launchpad.net/bugs/1450702) created by **Michael Millthorn (millthorn)**
```
Ok, updated Ubuntu (15.0.4 64 bit) today and the backups stopped working.

Backup Failed.

Failed with an unknown error.

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1500, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1494, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1343, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1476, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 631, in incremental_backup
    globals.backend)
  File "/usr/bin/duplicity", line 405, in write_multivol
    vi.set_hash("SHA1", gpg.get_hash("SHA1", tdp))
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 406, in
get_hash
    fp = path.open("rb")
  File "/usr/lib/python2.7/dist-packages/duplicity/path.py", line 551, in
open
    result = open(self.name, mode)
IOError: [Errno 2] No such file or directory: '/tmp/duplicity-iDn9gh-
tempdir/mktemp-AVmK4V-194'



----
org.gnome.DejaDup last-restore ''
org.gnome.DejaDup periodic true
org.gnome.DejaDup full-backup-period 90
org.gnome.DejaDup backend 'file'
org.gnome.DejaDup last-run '2015-04-29T23:43:29.114518Z'
org.gnome.DejaDup nag-check '2015-03-04T18:42:07.498008Z'
org.gnome.DejaDup prompt-check '2013-11-01T04:13:02.518780Z'
org.gnome.DejaDup root-prompt true
org.gnome.DejaDup include-list ['$HOME', '/home/michael/Data']
org.gnome.DejaDup exclude-list ['/home/michael/.local/share/Trash']
org.gnome.DejaDup last-backup '2015-04-29T23:43:29.114518Z'
org.gnome.DejaDup periodic-period 1
org.gnome.DejaDup delete-after 365
org.gnome.DejaDup.S3 id ''
org.gnome.DejaDup.S3 bucket ''
org.gnome.DejaDup.S3 folder 'ATS-LWS-001u'
org.gnome.DejaDup.GDrive email ''
org.gnome.DejaDup.GDrive folder '/deja-dup/ATS-LWS-001u'
org.gnome.DejaDup.Rackspace username ''
org.gnome.DejaDup.Rackspace container 'ATS-LWS-001u'
org.gnome.DejaDup.File path 'sftp://'
org.gnome.DejaDup.File short-name 'Backup'
org.gnome.DejaDup.File uuid '2CDEA1F0DEA1B30E'
org.gnome.DejaDup.File icon '. GThemedIcon drive-harddisk-usb drive-
harddisk drive'
org.gnome.DejaDup.File relpath b'LocalBackup'
org.gnome.DejaDup.File name 'TOSHIBA External USB 3.0: Backup'
org.gnome.DejaDup.File type 'volume'


---

Distributor ID: Ubuntu
Description:    Ubuntu 15.04
Release:        15.04
Codename:       vivid

---
```

Story points: 6.0
check_common_error is too permissive (lp:#1449057)

[Original report](https://bugs.launchpad.net/bugs/1449057) created by **David Coppit (coppit)**
```
I'm running rdiffdir, which is failing with errors like this:

Error ‘[Errno 17] File exists' processing .
Error '[Errno 1] Operation not permitted: './duplicity_temp.1'' processing
.

This bug report isn't about those. When this sort of failure happens, a
dupliciity_temp.# file is left around, and the program exits with a "0"
exit code. I suppose the reason is that these sorts of problems are hidden
by robust.check_common_error().

This failure mode is particularly nasty because the original (unpatched)
file is still there, which could cause one to think that it's patched when
it's not.

Can you change robust.check_common_error() to not hide IOErrors?
```

Story points: 6.0
Duplicity crashes when verifying (lp:#1445229)

[Original report](https://bugs.launchpad.net/bugs/1445229) created by **Mattias Månsson (mattias-mansson)**
```
Done a backup on my server, it's mostly photos. First I just did a full
backup of some folder, then added the rest with incremental. After
completion I tried a verify on my pictures folder just to check that it
worked. After a long time I get this error:

# duplicity verify --file-to-restore pub/pictures dpbx:/// /pub/pictures

Duplicity 0.6 series is being deprecated:
See http://www.nongnu.org/duplicity/

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Tue Apr 14 21:34:46 2015
GnuPG passphrase:
Traceback (most recent call last):
  File "/usr/local/bin/duplicity", line 1509, in <module>
    with_tempdir(main)
  File "/usr/local/bin/duplicity", line 1503, in with_tempdir
    fn()
  File "/usr/local/bin/duplicity", line 1352, in main
    do_backup(action)
  File "/usr/local/bin/duplicity", line 1439, in do_backup
    verify(col_stats)
  File "/usr/local/bin/duplicity", line 827, in verify
    for backup_ropath, current_path in collated:
  File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line
271, in collate2iters
    relem1 = riter1.next()
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line
499, in integrate_patch_iters
    for patch_seq in collated:
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line
379, in yield_tuples
    setrorps( overflow, elems )
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line
368, in setrorps
    elems[i] = iter_list[i].next()
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line
99, in filter_path_iter
    for path in path_iter:
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line
119, in difftar2path_iter
    multivol_fileobj.close() # aborting in middle of multivol
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line
239, in close
    if not self.addtobuffer():
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line
227, in addtobuffer
    self.tarinfo_list[0] = self.tar_iter.next()
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line
334, in next
    self.set_tarfile()
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line
323, in set_tarfile
    self.current_fp = self.fileobj_iter.next()
  File "/usr/local/bin/duplicity", line 729, in get_fileobj_iter
    backup_set.volume_name_dict[vol_num],
KeyError: 636

Duplicity 0.6.25
Python 2.7.3
Ubuntu Server 12.04.5 x86_64
Filesystem: ext4 on mdraid
```

Story points: 16.0
documentation to connect to European S3 bucket is wrong/misleading (lp:#1442571)

[Original report](https://bugs.launchpad.net/bugs/1442571) created by **az (az-debian)**
```
this is a forward of debian bug 782238, which lives here:
https://bugs.debian.org/782238

a user has complained that the documentation for using european s3 buckets
doesn't match reality. i don't use s3 so i have no idea whether this is a
valid issue or not.
```

Story points: 6.0
On successful restore duplicity exits with (-1) (lp:#1434041)

[Original report](https://bugs.launchpad.net/bugs/1434041) created by **Kenneth Loafman (kenneth-loafman)**
```
From: https://answers.launchpad.net/duplicity/+question/262087

We've got some automation built up around duplicity, but we're having some
occasional trouble with duplicity exiting with code -1 during a restore.
eg:

Writing foo/bar.jpg of type reg
Deleting /var/lib/tmp/duplicity-GYz7n0-tempdir/mktemp-yHEK5p-7
Processed volume 6 of 49
Ended with non-successful exit code (-1)

With the last line of output originating from our system.

I've looked at the pydoc for the ErrorCode class, and it explicitly states
that negative return codes are not to be used. The best I've been able to
turn up is that Python subprocesses will return a negative status
reflecting the signal they were sent, -1 correlating to SIGHUP.

Investigating this further I've found that duplicity.pexpect.run() appears
to be calling the close() function on child processes which issues a
SIGHUP, and then returning the child's status.

Is this an expected behaviour? Can we safely assume that a -1 exit status
is OK from duplicity?

This is pure inference, but is duplicity issuing an early close() on the
subprocess handling the restore once it has restored the necessary files,
and then the -1 return code gets passed up through the stack until
duplicity exits?
```

Story points: 8.0
Unknown responses not handled by duplicity (lp:#1432981)

[Original report](https://bugs.launchpad.net/bugs/1432981) created by **Samuel Bancal (samuel-bancal)**
```
As described in bug #1431019, we encountered a bug which led us to have no
incremental, but only full backups saved.

The solution has been to switch from scp:// to sftp:// , but this revealed
an unexpected behavior of duplicity: For several months, these backups
completed with exit code 0. Which led the admins to think the backups went
smoothly.

Could duplicity handle unknown responses in a manner that it returns an
error code?

Duplicity version : 0.7.02
Python version : 2.7.3
OS : Ubuntu 12.04.5 LTS
Target Filesystem : NFS through SSH (Ubuntu 12.04.5 LTS)
```

Story points: 6.0
Errno 84 Invalid or incomplete multibyte or wide character (lp:#1431394)

[Original report](https://bugs.launchpad.net/bugs/1431394) created by **dospaness (dospaness)**
```
I try to restore a backup to an external Hard-drive (USB) that already
contains the encrypted backup-files since I have no space on my internal
hardrive.

I have a similar issue as described here:

http://askubuntu.com/questions/448803/deja-dup-gives-invalid-or-incomplete-
multibyte-or-wide-character-when-attempti

and I can't figure out how to resolve it in order to restore my backup.

lsb_release -d
Description:    Ubuntu 14.04.2 LTS

dpkg-query -W deja-dup duplicity

deja-dup        30.0-0ubuntu4
duplicity       0.6.23-1ubuntu4.1

The debugging files come attached below. The error message that appears on
the command line is:

OSError: [Errno 84] Invalid or incomplete multibyte or wide character:
'/media/user/Backups_Envpolicies/restore_test_2015-03-06/home/user/2_data/geodata/climate/ibge/Zona
Econ\xf4mica Exclusiva _Lei.dbf'

It seems that deja dup has a problem to restore files that have a
unrecognized or invalid encoding. In my case it is group of files from
Brazil and the files came from MS Windows users. So it is probably encoded
in some strange way that nautlius does not recognize (please see the
screenshot attached).

So instead ofskipping the corrupt file and continuing to restore the rest,
deja dup stops the restoring process completly...
```

Story points: 12.0
gpg asks for password (lp:#1419618)

[Original report](https://bugs.launchpad.net/bugs/1419618) created by **Jon (jonablett)**
```
Using Duplicity 0.7.01

Using the --encrypt-key [key] option should avoid being prompted for the
password. I am getting a pinentry dialogue asking for the password (either
with or without --use-agent).

This might be related to a recent upgrade to GnuPG 2.1, not sure.

It doesn't matter if I enter a correct or incorrect password (or even
blank). It still proceeds to back up. My concern is that I want it to run
automatically in the background but I can't with pinentry stopping the
process.
```

Story points: 6.0
Misleading start time reported in statistics (lp:#1418543)

[Original report](https://bugs.launchpad.net/bugs/1418543) created by **Phill (phill.l)**
```
The statistics section reports a start and end time, however, the start
time on my machine is always over 5 minutes after the actual start time.
The section below shows a log file from a script that logs the current
time, immediately runs duplicity and then logs the end time. During the 5+
minutes not included in the statistics the CPU usage by duplicity (as shown
by `top`) is extremely high.

--

Backup Started Thu Feb 5 09:00:01 UTC 2015
Import of duplicity.backends.giobackend Failed: No module named gio
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Thu Jan 29 16:00:04 2015
--------------[ Backup Statistics ]--------------
StartTime 1423127161.48 (Thu Feb  5 09:06:01 2015)
EndTime 1423127165.40 (Thu Feb  5 09:06:05 2015)
ElapsedTime 3.92 (3.92 seconds)
SourceFiles 799
SourceFileSize 514518209 (491 MB)
NewFiles 0
NewFileSize 0 (0 bytes)
DeletedFiles 0
ChangedFiles 1
ChangedFileSize 1966187 (1.88 MB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 1
RawDeltaSize 14530 (14.2 KB)
TotalDestinationSizeChange 1467 (1.43 KB)
Errors 9
-------------------------------------------------

Backup Ended Thu Feb 5 09:06:06 UTC 2015

---

Package: duplicity
Status: install ok installed
Priority: optional
Section: utils
Installed-Size: 1028
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Version: 0.6.18-0ubuntu3.5
Depends: libc6 (>= 2.4), librsync1 (>= 0.9.6), python2.7, python (>=
2.7.1-0ubuntu2), python (<< 2.8), python-gnupginterface (>= 0.3.2-9.1),
python-lockfile
Suggests: python-boto, ncftp, rsync, ssh, python-paramiko
Breaks: deja-dup (<< 22.0-0ubuntu5)
Description: encrypted bandwidth-efficient backup
 Duplicity backs directories by producing encrypted tar-format volumes
 and uploading them to a remote or local file server. Because duplicity
 uses librsync, the incremental archives are space efficient and only
 record the parts of files that have changed since the last backup.
 Because duplicity uses GnuPG to encrypt and/or sign these archives, they
 will be safe from spying and/or modification by the server.
Homepage: http://duplicity.nongnu.org/
Original-Maintainer: Alexander Zangerl <az@debian.org>
```

Story points: 6.0
Option to specify changed files instead of scanning filesystem (lp:#1415621)

[Original report](https://bugs.launchpad.net/bugs/1415621) created by **Aaron Whitehouse (aaron-whitehouse)**
```
From Question #261358

Provide a way to pass a list of created/modified/deleted files to duplicity
and avoid duplicity doing a file system scan. This would speed up the
backup process.

This could take a list generated by fswatch (
http://emcrisostomo.github.io/fswatch/ ) as an input and would allow easier
creation of inotify-based backup systems (see, for example, Bug #781428 ).
```

Story points: 12.0
Update man pages for gdocs with example (lp:#1411894)

[Original report](https://bugs.launchpad.net/bugs/1411894) created by **ShadowXVII (thedarkestshadow)**
```
Currently the man pages use the following syntax as an example for gdocs.

gdocs://user[:password]@other.host/some_dir

It isn't clear that@ other.host should be substituted with@ gmail.com for
normal Google accounts, or@ domain.com for Google Apps accounts.

Google services usually require a full username (including@ gmail.com
suffix) when authenticating to other services, so it was natural to include
this in the user field. This causes parsing errors & two factor
authentication error messages.

The example should provide an explicit@ gmail.com example and show the
correct breakdown of user & password in relation to the syntax;

e.g. user bob@gmail.com with a password xyz into  the directory
/Backups/Duplicity;

gdocs://bob:xyz@gmail.com/Backups/Duplicity

not

gdocs://bob@gmail.com:xyz@other.host/some_dir

(Duplicity v0.7.0)
```
Original tags: man

Story points: 6.0