giseldo/neodataset huggingface dataset:
Crash when restoring data KeyError (lp:#498933)

[Original report](https://bugs.launchpad.net/bugs/498933) created by **Michael Terry (mterry)**
```
Binary package hint: deja-dup

1) Ubuntu 9.10
2) Deja Dup 10.2-0ubuntu1.1
3) I expected my backup to be restored
4) Deja Dup crashed instead.

I just tried to restore the backup I made, and it crashes repeatedly when
trying to restore.

I have it setup to backup my home directory, and tried restoring to both
the original location as well as a completely different folder. Both lead
to the same error message.


Traceback (most recent call last):
  File "/usr/bin/duplicity", line 825, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 818, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 775, in main
    restore(col_stats)
  File "/usr/bin/duplicity", line 436, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/usr/lib/python2.6/dist-packages/duplicity/patchdir.py", line 521,
in Write_ROPaths
    ITR(ropath.index, ropath)
  File "/usr/lib/python2.6/dist-packages/duplicity/lazy.py", line 336, in
__call__
    last_branch.fast_process, args)
  File "/usr/lib/python2.6/dist-packages/duplicity/robust.py", line 38, in
check_common_error
    return function(*args)
  File "/usr/lib/python2.6/dist-packages/duplicity/patchdir.py", line 574,
in fast_process
    ropath.copy(self.base_path.new_index(index))
  File "/usr/lib/python2.6/dist-packages/duplicity/path.py", line 412, in
copy
    other.writefileobj(self.open("rb"))
  File "/usr/lib/python2.6/dist-packages/duplicity/path.py", line 574, in
writefileobj
    buf = fin.read(_copy_blocksize)
  File "/usr/lib/python2.6/dist-packages/duplicity/patchdir.py", line 199,
in read
    if not self.addtobuffer():
  File "/usr/lib/python2.6/dist-packages/duplicity/patchdir.py", line 224,
in addtobuffer
    self.tarinfo_list[0] = self.tar_iter.next()
  File "/usr/lib/python2.6/dist-packages/duplicity/patchdir.py", line 331,
in next
    self.set_tarfile()
  File "/usr/lib/python2.6/dist-packages/duplicity/patchdir.py", line 320,
in set_tarfile
    self.current_fp = self.fileobj_iter.next()
  File "/usr/bin/duplicity", line 472, in get_fileobj_iter
    backup_set.volume_name_dict[vol_num],
KeyError: 15

ProblemType: Bug
Architecture: i386
Date: Sun Dec 20 19:59:06 2009
DistroRelease: Ubuntu 9.10
ExecutablePath: /usr/bin/deja-dup
InstallationMedia: Ubuntu 9.10 "Karmic Koala" - Release i386 (20091028.5)
Package: deja-dup 10.2-0ubuntu1.1
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-16.53-generic
SourcePackage: deja-dup
Uname: Linux 2.6.31-16-generic i686
```
Original tags: apport-bug i386

Story points: 84.0
Resuming an interrupted backup appears successful - but can’t restore from it (0.6.06) (lp:#491971)

[Original report](https://bugs.launchpad.net/bugs/491971) created by **Matthew Twomey (mtwomey)**
```
I'm doing backups using openssh scp/sftp to a remote host. When a full
backup is
disrupted in the middle of transferring and then restarted - Duplicity
starts on the next volume number *without* finishing the files that were
supposed to be in the previous volume.

That is to say:

*Normal non-disrupted backup*

Volume 1 = files 01 - 10
Volume 2 = files 11 - 20
Volume 3 = files 21 - 30

*Backup which was disrupted during network transfer of Volume 2 and then
restarted (Restarting after volume 2" was the message in the log)*

Volume 1 = files 01 - 10
Volume 2 = files 11 - 13 (or wherever it was disrupted)
Volume 3 = files 21 - 30

I've noticed that after a disruption, if I go in and delete the last volume
file (the one which was disrupted) on the remote location and then restart
the backup - it picks up with that volume, producing a usable backup. If I
don't delete the file, the restore process will error out when it get to
the volume that was disrupted.

Duplicity appears to expect temporary filenames to be used by the
OS/transfer software during transfer, which openssh does not use by
default.
```

Story points: 42.0
Restore fails with "Invalid data - SHA1 hash mismatch" (lp:#487720)

[Original report](https://bugs.launchpad.net/bugs/487720) created by **Andrew Fister (andrewfister)**
```
When restoring a backup, one might see an error like:

Invalid data - SHA1 hash mismatch:
Calculated hash: 0b2bc4c2fb98b36f9891f9172f909d70ab5662e9
Manifest hash: 11cd330357618de52e4e5361a6e63b09ee951ae2

This can happen when a volume file was not completely written to the
backend before duplicity was interrupted (say, shutting down the machine or
whatever).  When duplicity resumes the backup next run, it will start with
the next volume.  The half-complete volume file will sit on the backend and
cause this error later when restoring.

You can manually recover from this by either restoring from your older
backup sets or by restoring individual files that don't happen to be in the
corrupted volume.

== To Reproduce ==

See attachment
https://bugs.launchpad.net/ubuntu/+source/duplicity/+bug/487720/+attachment/2159465/+files/test.sh
for a test script tor reproduce the problem.

== Ubuntu SRU Justification ==

This is a serious data loss problem for users, which won't be noticed until
they try to restore.  With Ubuntu 11.10 including Deja Dup, some users may
think to back up their data first then upgrade, and may accidentally create
corrupted backups.
```
Original tags: verification-done-lucid verification-done-natty verification-needed

Story points: 204.0
error verbosity: include filename in output, make traceback less prominent (lp:#455081)

[Original report](https://bugs.launchpad.net/bugs/455081) created by **az (az-debian)**
```
i've got a backup of a large samba share which every now and then fails,
apparently because
of files being modified/truncated while duplicity is accessing them:
duplicity
dies with a permission denied on reading (all fair and square).

the bug here is that the exception info is extremely verbose wrt.
duplicity's calling stack but
completely lacks information useful to the user - like the name/path of the
file that causes the
problem.

the information given with the default verbosity level should at least
include the relevant information
(like the path here), and ideally not expose too much of the innards to an
end user - the traceback
isn't useful unless you're a duplicity developer and might be restricted to
higher verbosity levels.

here's the relevant output:
---
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Fri Oct  9 23:25:48 2009
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1241, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1234, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1216, in main
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 488, in incremental_backup
    globals.backend)
  File "/usr/bin/duplicity", line 295, in write_multivol
    globals.gpg_profile, globals.volsize)
  File "/usr/lib/python2.5/site-packages/duplicity/gpg.py", line 282, in
GPGWriteFile
    data = block_iter.next(min(block_size, bytes_to_go)).data
  File "/usr/lib/python2.5/site-packages/duplicity/diffdir.py", line 510,
in next
    result = self.process(self.input_iter.next(), size)
  File "/usr/lib/python2.5/site-packages/duplicity/diffdir.py", line 636,
in process
    data, last_block = self.get_data_block(fp, size - 512)
  File "/usr/lib/python2.5/site-packages/duplicity/diffdir.py", line 663,
in get_data_block
    buf = fp.read(read_size)
  File "/usr/lib/python2.5/site-packages/duplicity/diffdir.py", line 420,
in read
    buf = self.infile.read(length)
  File "/usr/lib/python2.5/site-packages/duplicity/diffdir.py", line 389,
in read
    buf = self.infile.read(length)
IOError: [Errno 13] Permission denied
---
```

Story points: 6.0
wishlist: differential backups (lp:#450885)

[Original report](https://bugs.launchpad.net/bugs/450885) created by **az (az-debian)**
```
this is a forward/copy of debian bug #550698 (which lives here:
http://bugs.debian.org/550698)
where the requester asks for differential backups to shorten the backup
chain.

the requester, however, also asks for full control over which earlier
backup
to use as base (for example a differential against an incremental backup)
which i don't believe makes much sense.

here's a copy of the request:
---cut---
I like how duplicity works, but having overlong backup chains to conserve
bandwidth/space puts too much pressure on the reliability of the remote
destination. I would like to keep the chains shorter, but can't due to
space
constraints as the only alternative is a periodic full backup.

Is there a reason as of why "incremental" could not accept a "-t" argument
to
specify which archive to use as the previous archive and let me manage the
storage by myself?

If I had a time or filename specifier like this, I could implement a
differential backup script on top of duplicity without any special handling
on
duplicity's part, much in the same way as you implement differential tar
archives.

As long as "restore" walks the backup chain from the most recent volume, it
should also work unchanged.

I've seen some old comments in the duplicity mailing list, but no proper
feature request. I'm also filing this here for other Debian users to see,
since the inability to perform differential archives kinds-of defeats the
initial space gain you have with duplicicy.

---cut---
```

Story points: 42.0
Missing volume file is not detected properly (lp:#447480)

[Original report](https://bugs.launchpad.net/bugs/447480) created by **Kenneth Loafman (kenneth-loafman)**
```
When I do a backup ( local folder to local folder, any options ) and delete
one of the archive files ( e.g. "duplicity-
full.20091008T074524Z.vol2.difftar.gpg" ) another call of duplicity will
add some files, but not the data that has been deleted before ( e.g. 25 MB
are lost ). This leaves an incomplete backup folder which doesn`t seem
right to me. After doing a backup with duplicity without it spitting out an
error message the backup archive is expected to be sane / complete.

```

Story points: 20.0
Allow restricting list-current-files (lp:#432104)

[Original report](https://bugs.launchpad.net/bugs/432104) created by **Michael Terry (mterry)**
```
When calling list-current-files, it would be nice if one could pass a
regexp for either files one is interested in or files one is not interested
in.  Maybe reuse the exclude/include arguments?

I could grep the output, but I suspect that's wasting time/resources to
list everything then exclude.
```

Story points: 20.0
Wishlist: Report size of backup (collection-status,list-files,...) (lp:#432092)

[Original report](https://bugs.launchpad.net/bugs/432092) created by **Michael Terry (mterry)**
```
When restoring, it would be nice to know the size of the backup.  I know
this is difficult because everything is gzipped up.  But I imagine either
there's a decent estimation that could be provided (the reported size
doesn't need to be 100% accurate, but hopefully errs on the side of over-
reporting), or duplicity could keep track via its metadata as incrementals
are added.

Once duplicity knows this, it can give an error if the destination is not
large enough to contain it.

Additionally, it would be nice if it was reported in, say, a collection-
status so that wrappers like Deja Dup could meaningfully use it.
```

Story points: 52.0
Add to deja-dup wake on lan functionality (lp:#431219)

[Original report](https://bugs.launchpad.net/bugs/431219) created by **emilio (emiliomaggio)**
```
Another item that I would like to add to the wish-list.

I have recently installed a low-power home server that is the target of my
backups. In general (As many users I guess) I am not at home for most of
the day and to save even more power (and the planet :-) ) I can set up the
server to go in S3 sleep mode if no processes are running and I can
manually wake-on-lan whenever I want the server up and running.

In case of duplicity I should not care to know when the backup is happening
and it would be great if duplicity could do the wake-up for me.
```

Story points: 40.0
incomplete chown() error msgs (lp:#415619)

[Original report](https://bugs.launchpad.net/bugs/415619) created by **az (az-debian)**
```
this is a forward/copy of debian bug #541788 (http://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=541788):
---
Package: duplicity
Version: 0.5.16-1~bpo50+1
Severity: normal

While running a test restore, duplicity reported these errors:

Error '[Errno 1] Operation not permitted: 'git/export'' processing export
Error '[Errno 1] Operation not permitted: 'git/hover.git/git-daemon-export-
ok''
processing hover.git/git-daemon-export-ok
Error '[Errno 1] Operation not permitted: 'git/irssi.git/git-daemon-export-
ok''
processing irssi.git/git-daemon-export-ok
Error '[Errno 1] Operation not permitted: 'git'' processing .

The files in question are owned by root on the source machine, the
duplicity
restore test was run on another machine using a non-root account.

I am guessing that the the reported failed operation is chown() as the
files
still belong to the user running the restore.

Duplicity should log which operation failed, including all details, e.g.
"setting owner to root:root" or similar.
---
```

Story points: 62.0
Support http_proxy (lp:#409739)

[Original report](https://bugs.launchpad.net/bugs/409739) created by **Michael Terry (mterry)**
```
Binary package hint: deja-dup

I'm using my own S3 server.  deja-dup does not support setting the S3
server
(i.e., it appears to be hard coded to use aws.amazon.com).  Generally this
is not
a problem if the program would respect the http proxy setting, however,
deja-dup
does not appear to do this.  When I run deja-dup as follows:

  http_proxy=http://localhost:8080 deja-dup

it fails and I do not see any access attempts in my S3 server's logs.

Thanks.

ProblemType: Bug
Architecture: i386
DistroRelease: Ubuntu 9.04
ExecutablePath: /usr/bin/deja-dup
Package: deja-dup 7.4-0ubuntu2
ProcEnviron:
 SHELL=/bin/bash
 PATH=(custom, user)
 LANG=en_US.UTF-8
SourcePackage: deja-dup
Uname: Linux 2.6.28-14-generic i686
```
Original tags: apport-bug i386

Story points: 14.0
On full drive, backup directory got deleted (lp:#1665327)

[Original report](https://bugs.launchpad.net/bugs/1665327) created by **Anes Lihovac (anes-lihovac-gmail)**
```
The other day, my harddrive containing my backups, was full, and of course
the backup triggered via dejadup would fail, which is configured to run
daily.  So the next day when the backup started again, I cancled it, only
to find out later that the directory containing my backups was empty. All
my backups were gone.
```

Story points: 6.0
Don't require email and password in hubic backend except for first time (lp:#1664063)

[Original report](https://bugs.launchpad.net/bugs/1664063) created by **Pablo Castellano (pablocastellano)**
```
When you configure Duplicity using Hubic backend you are required to create
~/.hubic_credentials with login data (email and password) and api data
(client_id, client_secret, redirect_uri.
More info here: http://duplicity.nongnu.org/duplicity.1.html#sect16

The first time you run duplicity, it will generate a ~/.hubic_tokens file
with two new values named access_token and refresh_token.

Since now you email and password are not used anymore but they are still
present in the configuration file. This is insecure because a malicious
user could read this file and compromise your whole hubic account.

Proposed workaround:
Once duplicity has obtained the tokens, set email and password to blank or
random data
```

Story points: 6.0
Restore fails with sha1 error.  Backups won't restore. (lp:#1661373)

[Original report](https://bugs.launchpad.net/bugs/1661373) created by **michaelcole (8-launchpad-michaelcole-com)**
```
This is similar to some bugs from 2009 which have fixes deployed in
Duplicity 0.6.

https://bugs.launchpad.net/deja-dup/+bug/487720

I saw GitLab melt down yesterday and wanted to check my backups would
restore.  I'm getting a sha1 error on restore.

ubuntu GNOME 16.04
duplicity 0.7.06
deja-dup 34.2
Python 2.7.12

Deja-Dup is configured to push backups to AWS.
I did a full backup and 4 incremental ones.

If I restore the most recent backup, it restores some files, then errors.
I pasted the error here, but don't know where the logs are.  It's
reproducible.  I won't fuss with it for a few days.

Invalid data - SHA1 hash mismatch for file:
 duplicity-inc.20170201T141348Z.to.20170202T133556Z.vol7.difftar.gpg
 Calculated hash: da39a3ee5e6b4b0d3255bfef95601890afd80709
 Manifest hash: d9adcb9246f4763e1d6345597b0f173cc4038f70

Hey, this is a big deal for me because backups that can't restore are worse
than no backups.
```

Story points: 6.0
Incorrect passphrase due to incorrect subkey use (lp:#1653406)

[Original report](https://bugs.launchpad.net/bugs/1653406) created by **Kapitan (aric81)**
```
Hello folks!

With the following on my Ubuntu machine:

duplicity 0.7.11
gpg (GnuPG) 2.1.11
libgcrypt 1.6.5

I am trying to execute:

/usr/bin/duplicity full -v4 --full-if-older-than 30D --exclude-filelist
~/.backup/HomeBackup.duplicity --gpg-options "--verbose --verify-options
no-show-photos" --encrypt-key=XXXXXXXX --sign-key=YYYYYYYY --exclude-
device-files /home pexpect+scp://user@host.com//home/user/backup/home

Note the different keys, neither X nor Y is a master key (both subkeys)

The problem is that none of my passphrase is working and the command always
fails:

===== Begin GnuPG log =====
gpg: using subkey YYYYYYYY instead of primary key MASTER-KEY
gpg: no default secret key: bad passphrase
gpg: [stdin]: sign+encrypt failed: bad passphrase
===== End GnuPG log =====

I investigated using gpg2 --edit and passwd and basically it seems that the
script uses the passphrase associated with the master key.

I am open to further investigation and thanks!
```
Original tags: gpg

Story points: 6.0
rdiff: Compute diffs in parallel (lp:#1652249)

[Original report](https://bugs.launchpad.net/bugs/1652249) created by **az (az-debian)**
```
this is a forward of debian bug 848950, which lives over there:
http://bugs.debian.org/848950

the original reporter requests an improvement to rdiffdir, namely to
parallelise diff computation.
```

Story points: 6.0
upload to S3 with multipart processing fails (lp:#1651430)

[Original report](https://bugs.launchpad.net/bugs/1651430) created by **Nick (n6ck)**
```
Hi,
we are using Amazon S3 to upload our backups with duplicity. Recently we
got more files to backup and the backup suddenly failed. I investigated and
found that it happened due to this bug
https://bugs.launchpad.net/duplicity/+bug/385495. To workaround this, I
changed our backup command to use multipart processing. My command line now
looks like this:

"""
env PASSPHRASE='' duplicity --log-file /var/log/duplicity/duplicity.log
--verbosity info --archive-dir=/var/tmp/duplicity --include-filelist <path-
to-include-file> --encrypt-key '0xC9BBFE6A' --encrypt-key '0x3247FB5E'
--encrypt-key '0xFA992415' --no-print-statistics --gpg-options '--trust-
model=always' --full-if-older-than 3D --allow-source-mismatch --s3-use-new-
style --s3-use-multiprocessing --s3-multipart-chunk-size 104857600 <base-
path> s3://s3-eu-central-1.amazonaws.com/<bucket-name>/<prefix>
"""

Sadly the upload of the first volume fails now with this output:

"""
Writing duplicity-full.20161220T121506Z.vol1.difftar.gpg
Uploading s3://s3-eu-central-1.amazonaws.com/<bucket>/<prefix>/duplicity-
full.20161220T121506Z.vol1.difftar.gpg to STANDARD Storage
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-
packages/duplicity/backends/_boto_multi.py", line 205, in _upload
    num_cb=max(2, 8 * bytes / (1024 * 1024))
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/multipart.py", line
260, in upload_part_from_file
    query_args=query_args, size=size)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 1225,
in set_contents_from_file
    fp.seek(0, os.SEEK_END)
  File "/usr/lib/python2.7/dist-packages/duplicity/filechunkio.py", line
47, in seek
    self.seek(self.bytes + offset)
  File "/usr/lib/python2.7/dist-packages/duplicity/filechunkio.py", line
43, in seek
    super(FileChunkIO, self).seek(self.offset + offset)
IOError: [Errno 22] Invalid argument
"""

I was using duplicity version '0.7.10-0ubuntu0ppa1240~ubuntu14.04.1', but
also tried the most current daily built version
'0.7.10-0ubuntu0ppa1240~ubuntu14.04.1'. We are using Ubuntu 14.04 with
python 2.7.6.

Can anyone help with this bug?
```

Story points: 14.0
[needs-packaging] MEGA Sync - web hosting (lp:#1650698)

[Original report](https://bugs.launchpad.net/bugs/1650698) created by **Szymon Scholz (quomoow)**
```
MEGA provides free 50GB per account space in his filehost.

URL: https://github.com/meganz/MEGAsync
License: Custom, Open Source
```
Original tags: needs-packaging

Story points: 6.0
Documentation: specify restore options (lp:#1644870)

[Original report](https://bugs.launchpad.net/bugs/1644870) created by **LKRaider (paul-eipper)**
```
In the man page and docs, it is not clear which options apply on
restoration vs backup.

The docs should specify which options can be used with each command.

Each command should have separate --help outputs.
```

Story points: 6.0
Feature request: restore resume (lp:#1644869)

[Original report](https://bugs.launchpad.net/bugs/1644869) created by **LKRaider (paul-eipper)**
```
Currently if duplicity restore stops for some reason, the only option is to
restart from beginning, which can take a long time (my connection drops
usually in the middle of the night, meaning restoring gets interrupted
every ~24h).

Add a feature such that calling the same duplicity restore command again
continues restoring the missing files on destination.


ENV:
$ duplicity --version
duplicity 0.7.10

$ python2.7 --version
Python 2.7.12

$ uname -a
Darwin 13.4.0 Darwin Kernel Version 13.4.0: Mon Jan 11 18:17:34 PST 2016;
root:xnu-2422.115.15~1/RELEASE_X86_64 x86_64
```

Story points: 8.0
The "restore-missing" functionality is not working as expected (lp:#1636255)

[Original report](https://bugs.launchpad.net/bugs/1636255) created by **Vej (vej)**
```
I have found out that the "--restore-missing" functionality is not working
as expected.
I have set up déja-dup to backup some folders every day.

If I delete some files today, and then I use the "restore-missing"
functionality the same day (today in my example), déja-dup shows me the
list of missing files as expected.

The problem is that if I use the "restore-missing" functionality from
tomorrow on, no files are listed anymore. It seems that "restore-missing"
seeks the missing files only if the backup is made the same day that I
deleted the files.
The behavior I expected is that "restore-missing" should search the entire
backup series, and create a list of all missing files since the first
backup was made.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: deja-dup 34.2-0ubuntu1
ProcVersionSignature: Ubuntu 4.4.0-45.66-generic 4.4.21
Uname: Linux 4.4.0-45-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
CurrentDesktop: Unity
Date: Mon Oct 24 18:09:40 2016
InstallationDate: Installed on 2014-04-23 (914 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64
(20140417)
SourcePackage: deja-dup
UpgradeStatus: Upgraded to xenial on 2016-07-31 (85 days ago)
```
Original tags: amd64 apport-bug third-party-packages xenial

Story points: 8.0