giseldo/neodataset huggingface dataset:
restore failed (Invalid data - SHA1 hash mismatch) (lp:#826389)

[Original report](https://bugs.launchpad.net/bugs/826389) created by **Manolis Kapernaros (kapcom01)**
```
I was using deja-dup to backup my home automatically every day on an NFS
folder with password encryption. Today I decided I had to format my HDD and
reinstall ubuntu so I tried to restore my files with deja-dup. The
restoring starts and I get back some of my files but suddenly it fails with
this error:

Invalid data - SHA1 hash mismatch:
Calculated hash: 7d96fd9b424f777e795c40f29f0cc88c7d87ec3d
Manifest hash: a102c548ef930c7a928f9519dbc5ea5f59441ebb

I tried to restore earlier backups but I get the same error.

I have really important files in these backups.. I'd appreciate if someone
can help me recover them. Thanks.

ProblemType: Bug
DistroRelease: Ubuntu 11.04
Package: deja-dup 18.1.1-0ubuntu1.1
ProcVersionSignature: Ubuntu 2.6.38-10.46-generic 2.6.38.7
Uname: Linux 2.6.38-10-generic i686
NonfreeKernelModules: nvidia wl
Architecture: i386
Date: Sun Aug 14 20:58:32 2011
EcryptfsInUse: Yes
InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Release i386 (20110427.1)
ProcEnviron:
 LANGUAGE=el_GR:en
 LANG=el_GR.UTF-8
 SHELL=/bin/bash
SourcePackage: deja-dup
UpgradeStatus: No upgrade log present (probably fresh install)
```
Original tags: apport-bug i386 natty running-unity

Story points: 30.0
Warning on initial backup unable to delete file (lp:#818547)

[Original report](https://bugs.launchpad.net/bugs/818547) created by **Dan Poirier (poirier)**
```
duplicity 0.6.14
Python 2.6.1
Mac OS X 10.6.8

Target filesystem local Mac file system

Command line:
rm -rf ~/.cache to
duplicity -v9 --no-encryption ./from file://`pwd`/to  >dup.out 2>&1

Seeing these messages with default verbosity:
Unable to delete
/Users/poirier/.cache/duplicity/02164a79c531c573207470764797d0af/duplicity-
full-signatures.20110730T150346Z.sigtar.gz: [Errno 2] No such file or
directory:
'/Users/poirier/.cache/duplicity/02164a79c531c573207470764797d0af/duplicity-
full-signatures.20110730T150346Z.sigtar.gz'
Writing /Users/poirier/tmp/to/duplicity-full.20110730T150346Z.manifest
Unable to delete
/Users/poirier/.cache/duplicity/02164a79c531c573207470764797d0af/duplicity-
full.20110730T150346Z.manifest: [Errno 2] No such file or directory:
'/Users/poirier/.cache/duplicity/02164a79c531c573207470764797d0af/duplicity-
full.20110730T150346Z.manifest'

Attaching full output.
```

Story points: 22.0
Passphrase requested more than once (lp:#816954)

[Original report](https://bugs.launchpad.net/bugs/816954) created by **chrispoole (chris-chrispoole)**
```
When asked for a passphrase for a sign key, Duplicity requests is twice, to
ensure it has been entered correctly.

In line with other unix cli programs, where the user is assumed to know
exactly what he's doing, it should be requested only once.

The worst thing that could happen would be for gpg to reject the
passphrase, and have to start Duplicity again.

duplicity 0.6.14.
```

Story points: 8.0
tmpdir error with encryption (lp:#812278)

[Original report](https://bugs.launchpad.net/bugs/812278) created by **andreas owen (aowen)**
```
i am trying to make a encrypted backup but am getting wierd results. No
files are writen in the destination folder and the log file is inconclusiv.
Unencrypted backups work fine. Can someone please help?

I have attached the log file, output of gpg2 --list-key an the shell script
I use. This is about the only duplicity full backup that is active in the
shell script.

this is the terminal output:
<code>
owiLand> sh /etc/duplicity/duplicity_backup.sh
Traceback (most recent call last):
  File "/opt/bin/duplicity", line 1250, in <module>
    with_tempdir(main)
  File "/opt/bin/duplicity", line 1243, in with_tempdir
    fn()
  File "/opt/bin/duplicity", line 1216, in main
    full_backup(col_stats)
  File "/opt/bin/duplicity", line 417, in full_backup
    globals.backend)
  File "/opt/bin/duplicity", line 295, in write_multivol
    globals.gpg_profile, globals.volsize)
  File "/opt/lib/python2.6/site-packages/duplicity/gpg.py", line 275, in
GPGWriteFile
    bytes_to_go = data_size - get_current_size()
  File "/opt/lib/python2.6/site-packages/duplicity/gpg.py", line 267, in
get_current_size
    return os.stat(filename).st_size
OSError: [Errno 2] No such file or directory: '/tmp/duplicity-
zrwlI0-tempdir/mktemp-_gifDQ-2'

sending incremental file list

sent 64 bytes  received 12 bytes  152.00 bytes/sec
total size is 2310082  speedup is 30395.82
</code>

i have a ds211+ with the newest firmeware and duplicity installed, target
of file system it ext4.
```

Story points: 16.0
duplicity doesn't work with --use-agent when the key hasn't been loaded yet (lp:#811218)

[Original report](https://bugs.launchpad.net/bugs/811218) created by **Peter Simons (simons-s)**
```
I'd like duplicity to use my gpg-agent. When I run it before my key is
known to the agent, however, it just aborts with an error. Instead, I would
like the agent to pop up the "pinentry" window on that occasion, so that I
can enter the passphrase.

I'm using duplicity 0.6.14.
```

Story points: 6.0
Python version defaults to /usr/bin/python, should be /usr/bin/env python (lp:#804803)

[Original report](https://bugs.launchpad.net/bugs/804803) created by **Dan Loewenherz (dan-wp)**
```
I wrestled with an ImportError for a bit until I realized that Python
wasn't being set the python version first found in PATH. Changing the first
line of the main duplicity script from /usr/bin/python to /usr/bin/env
python should fix the issue.
```
Original tags: python

Story points: 6.0
Unable to connect to Rackspace UK servers (lp:#793997)

[Original report](https://bugs.launchpad.net/bugs/793997) created by **Tor Inge Schulstad (tor-inge-schulstad)**
```
I have an acconunt at Rackspace, but I use the London servers. I log on to
https://lon.manage.rackspacecloud.com
I'm not able to use this account to store my backups. I only get
AuthenticationFailed.

Software versions:
deja-dup        18.1.1-0ubuntu1
duplicity               0.6.13-0ubuntu1

From logfile:
DUPLICITY: INFO 1
DUPLICITY: . Import of duplicity.backends.webdavbackend Succeeded
DUPLICITY: ERROR 38
DUPLICITY: . Connection failed, please check your credentials:
AuthenticationFailed

Settings:
org.gnome.DejaDup backend 'rackspace'
org.gnome.DejaDup delete-after 56
org.gnome.DejaDup encrypt true
org.gnome.DejaDup exclude-list ['$TRASH', '$DOWNLOAD']
org.gnome.DejaDup include-list ['/media/WD_ELEMENTS/bilder/2011']
org.gnome.DejaDup last-run ''
org.gnome.DejaDup periodic true
org.gnome.DejaDup periodic-period 14
org.gnome.DejaDup root-prompt true
org.gnome.DejaDup.File icon ''
org.gnome.DejaDup.File name ''
org.gnome.DejaDup.File path ''
org.gnome.DejaDup.File relpath@ ay []
org.gnome.DejaDup.File short-name ''
org.gnome.DejaDup.File type 'normal'
org.gnome.DejaDup.File uuid ''
org.gnome.DejaDup.Rackspace container 'dejadup'
org.gnome.DejaDup.Rackspace username 'tischulstad'
org.gnome.DejaDup.S3 bucket ''
org.gnome.DejaDup.S3 folder 'workstation'
org.gnome.DejaDup.S3 id ''
org.gnome.DejaDup.U1 folder '/deja-dup/$HOSTNAME'

Distro: Ubuntu 11.04
```

Story points: 30.0
AssertionError after incomplete incremental backup (lp:#784098)

[Original report](https://bugs.launchpad.net/bugs/784098) created by **Jens Finkhäuser (finkhaeuser-consulting)**
```
Duplicity version: 0.6.08b
Python version: 2.6.5
OS Distro and version: Ubuntu 10.04 server

Local and Remote metadata are synchronized, no sync needed.
Last inc backup left a partial set, restarting.
Last full backup date: Thu Mar 10 04:47:43 2011
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1239, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1232, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1214, in main
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 474, in incremental_backup
    assert dup_time.curtime != dup_time.prevtime, "time not moving forward
at appropriate pace - system clock issues?"
AssertionError: time not moving forward at appropriate pace - system clock
issues?

Looking at the code, duplicity seems to sleep just before line 474, and
then requests curtime again. It's not clear how that value should have
changed in the meantime; the value returned is from the global curtime
variable in dup_time.py.

Without understanding the rest of the code further, this looks like a bug
in the setcurtime() function; clearly the line "t = time_in_secs or curtime
or long(time.time()) # only set from NOW once" fails to set the current
time from NOW in this instance, which is presumably what is expected.
```

Story points: 6.0
Don't write new incremental set if nothing changed (lp:#768481)

[Original report](https://bugs.launchpad.net/bugs/768481) created by **Adam Porter (alphapapa)**
```
I have a script that calls Duplicity separately on smaller sets of data so
that important ones get backed up first.  Some of those early sets rarely
change, but even if nothing changes, Duplicity uploads a new incremental
set of files.  This not only wastes time when backing up, but it causes a
LOT of time to be wasted when doing a restore.

For example, a few days ago I needed to restore three small files from a
small backup set that hadn't changed in years.  But to do so, Duplicity had
to download probably over a hundred incremental sets in which nothing
changed.  This took a LONG time!  It's especially slow since, judging from
Duplicity's verbose output, it downloads one file at a time--the latency
between commands adds up quickly (I'm ignorant about SFTP: is there no way
to request more than one file at a time?).  Restoring less than 20 KB of
data took over 10 minutes because of all the empty incremental sets that
had to be downloaded.

I realize that it's important to indicate when incremental backups were
made, even if nothing changed, so I think Duplicity should store empty
incremental sets in a different way.  For example, it could store a single
file in the remote directory that lists all the times incremental backups
were made in which nothing changed.  When making a new, unchanged
incremental backup, Duplicity could simply append the date and time to the
file and replace the copy on the server.  This would save so much time it's
not even funny!  :)
```

Story points: 58.0
duplicity should fail gracefully on empty GPG signature file (lp:#764301)

[Original report](https://bugs.launchpad.net/bugs/764301) created by **Olivier Berger (oberger)**
```
(This is a copy of http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=623184)

Here's what I get with an empty remote GPG file (probably resulting of out
of quota), when trying to resync the local cache :

...
Fetching .../duplicity-
inc.20101225T001425Z.to.20101226T001506Z.manifest.gpg to /tmp/duplicity-
mqzqB7-tempdir/mktemp-f0fGqJ-7'
State = sftp, Before = 'quit'
Removing still remembered temporary file /tmp/duplicity-
mqzqB7-tempdir/mkstemp-xGZcgl-1
Removing still remembered temporary file /tmp/duplicity-
mqzqB7-tempdir/mktemp-f0fGqJ-7
GPG error detail: Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1251, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1244, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1145, in main
    sync_archive()
  File "/usr/bin/duplicity", line 959, in sync_archive
    copy_to_local(fn)
  File "/usr/bin/duplicity", line 915, in copy_to_local
    globals.archive_dir.append(loc_name).name)
  File "/usr/bin/duplicity", line 841, in copy_raw
    data = src_iter.next(block_size).data
  File "/usr/bin/duplicity", line 900, in next
    self.fileobj.close()
  File "/usr/lib/python2.6/dist-packages/duplicity/dup_temp.py", line 210,
in close
    assert not self.fileobj.close()
  File "/usr/lib/python2.6/dist-packages/duplicity/gpg.py", line 198, in
close
    self.gpg_failed()
  File "/usr/lib/python2.6/dist-packages/duplicity/gpg.py", line 165, in
gpg_failed
    raise GPGError, msg
GPGError: GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: decrypt_message failed: eof
===== End GnuPG log =====


GPGError: GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: decrypt_message failed: eof
===== End GnuPG log =====

Of course, by listing the cache one can find the culprit 0 size file :
duplicity-inc.20101225T001425Z.to.20101226T001506Z.manifest.gpg

But I guess it wouldn't be so hard to have a check against empty files and
manage the failing backup/restore then.

Hope this helps.

Best regards,
```

Story points: 14.0
non-interactive duplicity job using GPG keys crashes (lp:#758077)

[Original report](https://bugs.launchpad.net/bugs/758077) created by **Joshua Jensen (joshua-joshuajensen)**
```
I'm running a duplicity nightly backup from cron that is having problems.
I'm using a GPG keypair.  The public key is used to encrypt without a
PASSPHRASE or interaction, and the private key to decrypt and restore
interactively with a PASSPHRASE later if necessary.

This works from my command prompt  *where I am not prompted for anything
when running this command*.

However, from cron, this fails with getpass.py being uncomfortable about
not being able to control echo on the terminal:

ARGS="--asynchronous-upload --tempdir /tmp --no-print-statistics \
        --include /data/backups/bin --encrypt-key CE71E9DC \
        --exclude-globbing-filelist $DEST/bin/rdiff-backup-excludes \
        -v2 --volsize 100 $@"

duplicity $ARGS / file://$DESTDIR


/usr/lib64/python2.6/getpass.py:83: GetPassWarning: Can not control echo on
the terminal.
  passwd = fallback_getpass(prompt, stream)
Warning: Password input may be echoed.
GnuPG passphrase: Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1245, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1238, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1139, in main
    sync_archive()
  File "/usr/bin/duplicity", line 949, in sync_archive
    globals.gpg_profile.passphrase = get_passphrase(1, "sync")
  File "/usr/bin/duplicity", line 129, in get_passphrase
    pass1 = getpass.getpass("GnuPG passphrase: ")
  File "/usr/lib64/python2.6/getpass.py", line 83, in unix_getpass
    passwd = fallback_getpass(prompt, stream)
  File "/usr/lib64/python2.6/getpass.py", line 118, in fallback_getpass
    return _raw_input(prompt, stream)
  File "/usr/lib64/python2.6/getpass.py", line 135, in _raw_input
    raise EOFError
EOFError

duplicity version 0.6.11, Python 2.6.5, RHEL 6.0 x86_64, target filesystem
is an NFS mount




Surely non-interactive duplicity commands shouldn't fail because duplicity
can't go interactive!!
```

Story points: 18.0
Device numbers overflow on Linux x86_64 (lp:#755583)

[Original report](https://bugs.launchpad.net/bugs/755583) created by **Nate Eldredge (nate-thatsmathematics)**
```
In trying to use Duplicity to archive some backups from another OS, I found
that it fails on device files with certain major/minor numbers.  This is on
Linux Ubuntu maverick x86_64; I cannot reproduce it on 32-bit x86.  I've
attached a log with the error message.

This is Duplicity 0.6.13, using the ubuntu package
0.6.13-0ubuntu1~maverick1.  Python is 2.6.6, using the ubuntu package
2.6.6-2ubuntu2.  The filesystem is ext4.

It seems this happens when the device file's st_rdev exceeds 2^31.  The
major and minor numbers are packed into this field in a funny way so it
takes a little thought to see how to reproduce this.  The encoding goes as
follows: the 32 bits of st_rdev from high to low are:

MIN[19:8] MAJ[11:0] MIN[7:0]

(Actually it looks like st_rdev is really 64 bits, encoded as MAJ[31:12]
MIN[31:8] MAJ[11:0] MIN[7:0], but it is not currently possible to create a
device with the high 32 bits nonzero.)

So we have a 12-bit major number and a 20-bit minor number.  It appears
duplicity fails when bit 20 of the minor number is set.  E.g. 'mknod foo c
0xfff 0xfffff' or 'mknod foo c 0x000 0x80000'.

Using the --exclude-device-files options does NOT prevent duplicity from
failing.
```

Story points: 6.0
duplicity crashes when attempted to read a "write only" file (lp:#750570)

[Original report](https://bugs.launchpad.net/bugs/750570) created by **Joshua Jensen (joshua-joshuajensen)**
```
When doing a backup of my machine, duplicity dies on reading in the files
in /selinux

This is not a fake or mounted filesystem like say /sys or /proc, it is a
real ext4 fs:

$ ls -l /selinux/
total 0
-rw-rw-rw-.  1 root root    0 Mar 17 16:59 access
dr-xr-xr-x.  2 root root    0 Mar 17 16:59 avc
dr-xr-xr-x.  2 root root    0 Mar 17 16:59 booleans
-rw-r--r--.  1 root root    0 Mar 17 16:59 checkreqprot
dr-xr-xr-x. 79 root root    0 Mar 17 16:59 class
--w-------.  1 root root    0 Mar 17 16:59 commit_pending_bools
-rw-rw-rw-.  1 root root    0 Mar 17 16:59 context
-rw-rw-rw-.  1 root root    0 Mar 17 16:59 create
-r--r--r--.  1 root root    0 Mar 17 16:59 deny_unknown
--w-------.  1 root root    0 Mar 17 16:59 disable
-rw-r--r--.  1 root root    0 Mar 17 16:59 enforce
dr-xr-xr-x.  2 root root    0 Mar 17 16:59 initial_contexts
-rw-------.  1 root root    0 Mar 17 16:59 load
-rw-rw-rw-.  1 root root    0 Mar 17 16:59 member
-r--r--r--.  1 root root    0 Mar 17 16:59 mls
crw-rw-rw-.  1 root root 1, 3 Mar 17 16:59 null
dr-xr-xr-x.  2 root root    0 Mar 17 16:59 policy_capabilities
-r--r--r--.  1 root root    0 Mar 17 16:59 policyvers
-r--r--r--.  1 root root    0 Mar 17 16:59 reject_unknown
-rw-rw-rw-.  1 root root    0 Mar 17 16:59 relabel
-rw-rw-rw-.  1 root root    0 Mar 17 16:59 user

Notice the "write-only" files... which duplicity can't handle:


A selinux/class/x_synthetic_event/perms/receive
A selinux/class/x_synthetic_event/perms/send
A selinux/commit_pending_bools
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1245, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1238, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1216, in main
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 417, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 295, in write_multivol
    globals.gpg_profile, globals.volsize)
  File "/usr/lib64/python2.6/site-packages/duplicity/gpg.py", line 279, in
GPGWriteFile
    data = block_iter.next(min(block_size, bytes_to_go)).data
  File "/usr/lib64/python2.6/site-packages/duplicity/diffdir.py", line 505,
in next
    result = self.process(self.input_iter.next(), size)
  File "/usr/lib64/python2.6/site-packages/duplicity/diffdir.py", line 631,
in process
    data, last_block = self.get_data_block(fp, size - 512)
  File "/usr/lib64/python2.6/site-packages/duplicity/diffdir.py", line 658,
in get_data_block
    buf = fp.read(read_size)
  File "/usr/lib64/python2.6/site-packages/duplicity/diffdir.py", line 415,
in read
    buf = self.infile.read(length)
  File "/usr/lib64/python2.6/site-packages/duplicity/diffdir.py", line 384,
in read
    buf = self.infile.read(length)
IOError: [Errno 22] Invalid argument



Can we get a fix for this?  Probably just ignoring these files makes sense.
```

Story points: 6.0
reference backups  by unique IDs (not url or name) (lp:#743820)

[Original report](https://bugs.launchpad.net/bugs/743820) created by **ceg (ceg)**
```
The idea is to store the UUID in the backup files, and use it for the name
of the cache dir.
Then the correct cache should always be identifyable.
Backup --name option would be stored with the backup files and in the
cache.
Time of last backups, URLs etc. could be cached for integrity,
informational or convenience reasons.

If duplicity full/incremental/restore is called only with the --name (or an
--ID) option, duplicity could use the cached URL(s) and settings.

* The cache name would not contain a hash of the url, and stay valid if url
changes.
* Named backups could be renamed without breaking older copies.
...
```

Story points: 14.0
New feature: Exclude files based on size (lp:#743725)

[Original report](https://bugs.launchpad.net/bugs/743725) created by **jlh (jlherren)**
```
I'd really like to exclude large files from my backups. I sometimes have
huge SQL dumps laying around and they take hours (if not days) to transfer
to my backup host.

Something like --exclude-size or similar. Don't know if an --include-size
counterpart makes any sense.
```

Story points: 8.0
remove-all-but-n-full does not check for successfull backup (lp:#730497)

[Original report](https://bugs.launchpad.net/bugs/730497) created by **Stefan Voelkel (stefan-voelkel)**
```
duplicity 0.6.09
Python 2.5.2
Debian Lenny
Target: Linxux, via SSH, no encryption

My backup script contains:

  duplicity --exclude-if-present .duplicity.nobackup --include ...
--exclude '**' --archive-dir ... / ssh://...//mnt/duplicity/ --no-
encryption --full-if-older-than 180D
  duplicity remove-all-but-n-full 1 ssh://...//mnt/duplicity/ --no-
encryption --force

The last full backup was more than 180 days old, so duplicity created a new
full backup. However the filesystem was too small, so the new full backup
was incomplete.

It seems that the call to remove-all-but-n-full does not check, if the last
n full backups were successfull, as it removed the old (complete) full
backup, and kept the new (broken) full backup:

$ ls -s
total 529732
    0 duplicity-full.20110306T114016Z.manifest
25612 duplicity-full.20110306T114016Z.vol10.difftar.gz
...
17808 duplicity-full.20110306T114016Z.vol21.difftar.gz
    0 duplicity-full.20110306T114016Z.vol22.difftar.gz
    0 duplicity-full.20110306T114016Z.vol23.difftar.gz
25588 duplicity-full.20110306T114016Z.vol2.difftar.gz
...
25592 duplicity-full.20110306T114016Z.vol9.difftar.gz
    0 duplicity-full-signatures.20110306T114016Z.sigtar.gz
```

Story points: 20.0