giseldo/neodataset huggingface dataset:
rdiffdir: Broken sigs for newly created files (lp:#1387786)

[Original report](https://bugs.launchpad.net/bugs/1387786) created by **David Coppit (coppit)**
```
This is potentially a very bad bug. It causes rdiffdir to create invalid
signature files, and the resulting delta files are full of zeroes.

1) Create a shell script with the commands below.

2) Run it. You should see "Done"

3) Instead you see:

Delta doesn't appear valid
Files FILE1 and FILE2 differ
Delta did not properly patch FILE
Done

I reproduced this on CentOS 6.4 and OS X 10.10 (Yosemite).

Some observations:
* If you run the commands one by one manually it works
* The sig file seems to be the problem
* If you create a larger file, say 10MB using dd, it will succeed perhaps 1
in 20 times. Otherwise this script only succeeds about 1 in 7000 times.

I could speculate that librsync is using a lower-level file access API that
is bypassing OS file caches. Let me know if I need to file this bug with
librsync instead.

RDIFFDIR=rdiffdir

rm -f FILE1 FILE2
echo 0123456789 > FILE1
echo xxxxxxxxxx > FILE2

#rm -f FILE1 FILE2
#dd if=/dev/urandom of=FILE1 bs=10M count=1 2>/dev/null
#cp FILE1 FILE2
#echo "a" >> FILE2

rm -f FILE1.sig
$RDIFFDIR sig FILE1 FILE1.sig

rm -f FILE2.delta
$RDIFFDIR delta FILE1.sig FILE2 FILE2.delta

xxd FILE2.delta | grep diff -q || echo "Delta doesn't appear valid"

$RDIFFDIR patch FILE1 FILE2.delta

diff -q FILE1 FILE2 || echo "Delta did not properly patch FILE"

echo "Done"
```
Original tags: librsync rdiffdir

Story points: 6.0
missing python-oauth (lp:#1382215)

[Original report](https://bugs.launchpad.net/bugs/1382215) created by **rdesfo (rdesfo)**
```
there is a bug in the deja-dup repo that is resolved by installing [python-
oauth](https://bugs.launchpad.net/deja-dup/+bug/1184225).

Should 'python-oauth' be added as a dependency to duplicity?
```

Story points: 6.0
rdiffdir: Support user-specified output dir for better multi-diff performance (lp:#1381754)

[Original report](https://bugs.launchpad.net/bugs/1381754) created by **David Coppit (coppit)**
```
Here's the use case that I'm using rdiffdir for:

1) Create a base VM and distribute it
2) Create version 1 of the VM from the base, compute binary diff,
distribute it
3) On remote hosts, copy the base VM, and apply the diff
4) Create version 2 of the VM from the base, compute the binary diff,
distribute it
5) On remote hosts, copy the base VM, and apply the diff

So these are not incremental diffs, but rather all based off the "base" VM.
On the remote hosts, the following is happening:

1) copy VM image from cache to target dir
2) rdiffdir copies the VM image from target dir to temp file, then patches
it

So there's an extra copy in there that is killing my performance. It's
actually faster to download a full VM image (one full write) instead of
downloading the patch and patching the previously downloaded image (2 full
reads, 1 full write, 1 delta write)

What I'd like to be able to do is have rdiffdir allow me to specify an
output directory for the patched files. The default would be as it is
today, which is to patch the same directory as the input. But with the
option, one could leave the original files untouched, and store the patched
files into the output directory.

duplicity version 0.6.24
python 2.7.5
Any OS
```
Original tags: rdiffdir

Story points: 6.0
rdiffdir: pipe STDIN doesn't work. Docs unclear (lp:#1378986)

[Original report](https://bugs.launchpad.net/bugs/1378986) created by **David Coppit (coppit)**
```
The man page says this:

       rdiffdir [options] sig[nature] basis_dir signature_file

       rdiffdir [options] delta full_sigtar {incr_sigtar} new_dir
delta_file

       rdiffdir [options] patch basis_dir delta_file

       rdiffdir [options] tar basis_dir tar_file

       If signature_file or delta_file are "-", the data will be read from
stdin or written to stdout as appropriate.

The last sentence is confusing, because neither signature_file or
delta_file are inputs. Perhaps the list should include full_sigtar? Or
maybe full_sigtar should be renamed to signature_file?

I don't know if this should be a separate bug, but it's related. This
doesn't work:

$ rdiffdir sig . /tmp/file.sig

$ cat /tmp/file.sig | rdiffdir delta - . /tmp/file.delta
Traceback (most recent call last):
  File "/usr/local/Cellar/duplicity/0.6.24/libexec/bin/rdiffdir", line 232,
in <module>
    main()
  File "/usr/local/Cellar/duplicity/0.6.24/libexec/bin/rdiffdir", line 217,
in main
    write_delta(file_args[-2], sig_infp, delta_outfp)
  File "/usr/local/Cellar/duplicity/0.6.24/libexec/bin/rdiffdir", line 179,
in write_delta
    diffdir.write_block_iter(delta_iter, outfp)
  File "/usr/local/Cellar/duplicity/0.6.24/lib/python2.7/site-
packages/duplicity/diffdir.py", line 719, in write_block_iter
    for block in block_iter:
  File "/usr/local/Cellar/duplicity/0.6.24/lib/python2.7/site-
packages/duplicity/diffdir.py", line 518, in next
    result = self.process(self.input_iter.next())
  File "/usr/local/Cellar/duplicity/0.6.24/lib/python2.7/site-
packages/duplicity/diffdir.py", line 190, in get_delta_iter
    for new_path, sig_path in collated:
  File "/usr/local/Cellar/duplicity/0.6.24/lib/python2.7/site-
packages/duplicity/diffdir.py", line 281, in collate2iters
    relem2 = riter2.next()
  File "/usr/local/Cellar/duplicity/0.6.24/lib/python2.7/site-
packages/duplicity/diffdir.py", line 346, in combine_path_iters
    range(len(path_iter_list))))
  File "/usr/local/Cellar/duplicity/0.6.24/lib/python2.7/site-
packages/duplicity/diffdir.py", line 322, in get_triple
    path = path_iter_list[iter_index].next()
  File "/usr/local/Cellar/duplicity/0.6.24/lib/python2.7/site-
packages/duplicity/diffdir.py", line 232, in sigtar2path_iter
    tf = util.make_tarfile("r", sigtarobj)
  File "/usr/local/Cellar/duplicity/0.6.24/lib/python2.7/site-
packages/duplicity/util.py", line 113, in make_tarfile
    tf = tarfile.TarFile("arbitrary", mode, fp)
  File "/usr/local/Cellar/duplicity/0.6.24/lib/python2.7/site-
packages/duplicity/tarfile.py", line 1565, in __init__
    self.offset = self.fileobj.tell()
IOError: [Errno 29] Illegal seek

But this does:

$ rdiffdir delta - . /tmp/file.delta < /tmp/file.sig
```
Original tags: rdiffdir

Story points: 6.0
rdiffdir RFE: Support omitting timestamp in gzip'd signature and delta files (lp:#1377959)

[Original report](https://bugs.launchpad.net/bugs/1377959) created by **David Coppit (coppit)**
```
In rdiffdir, the -z option produces signature and delta files that have an
embedded timestamp. gzip has the --no-name option, which will omit the
timestamp and some other info, so that the resulting file is predictably
the same.

The reason this is important is that people may run md5sum on the signature
or delta in order to see if it's been generated before. Timestamps make
that impossible.

I looked at the file after running gzip --no-name, and it appears that gzip
inserts a "0" for the time. So in python, it looks like you can simply pass
"mtime=0" to the gzip.GzipFile() function. I would submit a patch, but I'm
not sure if you want this to be optional, or if duplicity requires the
embedded timestamp for some reason.

The workaround is to run "gzip --no-name" on the files after running
rdiffdir.

$ rdiffdir -V
rdiffdir 0.6.24

$ python --version
Python 2.7.5

$ uname -a
Darwin dhcp-10-20-61-194.nvidia.com 13.3.0 Darwin Kernel Version 13.3.0:
Tue Jun  3 21:27:35 PDT 2014; root:xnu-2422.110.17~1/RELEASE_X86_64 x86_64
i386 MacBookPro10,1 Darwin
```
Original tags: rdiffdir

Story points: 6.0
not found in archive, no files restored (lp:#1373610)

[Original report](https://bugs.launchpad.net/bugs/1373610) created by **mancu (mancurian)**
```
I have been backing up for a while now I need a file and having a classic
issue which is the back up tools stopping when we really need them.

I am trying to restore a file and getting this

duplicity -v4 -r xxx/xxxxxx file:///cygdrive/v/xxxxx/DUPLICITY/xxxxxxx/ del


Error 'basis_file must be a (true) file' patching . xxx/xxxxx not found in
archive, no files restored.


I tried it on linux and cygwin, same issue.
```

Story points: 6.0
Duplicity 0.6.24 doesn't show statistics after backup (lp:#1373327)

[Original report](https://bugs.launchpad.net/bugs/1373327) created by **sander eikelenboom (b-linux)**
```
After upgrading duplicity on my debian systems i noticed that the backup
statistics that were printed after a backup by default are not there
anymore. The commandline with which duplicity is invoked hasn't changed.

Upgrade to 0.6.24 (backport package Debian wheezy)

Before it used to print:
-----------[Incremental]------------
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sun Sep 21 01:30:18 2014
--------------[ Backup Statistics ]--------------
StartTime 1411515020.44 (Wed Sep 24 01:30:20 2014)
EndTime 1411515020.49 (Wed Sep 24 01:30:20 2014)
ElapsedTime 0.05 (0.05 seconds)
SourceFiles 1
SourceFileSize 4096 (4.00 KB)
NewFiles 0
NewFileSize 0 (0 bytes)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 0
RawDeltaSize 0 (0 bytes)
TotalDestinationSizeChange 103 (103 bytes)
Errors 0
-------------------------------------------------

After it only prints:
-----------[Incremental]------------
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sun Sep 21 01:30:18 2014

The manpage still says:
       --no-print-statistics
              By default duplicity will print statistics about the current
session after a successful backup.  This switch disables that  behavior.

So i would expect them to be still printed by default :)
```

Story points: 6.0
Feature Request: Option to forgo compression (or ability to specify tar options) (lp:#1369499)

[Original report](https://bugs.launchpad.net/bugs/1369499) created by **Fjodor (sune-molgaard)**
```
Hiya,

I do duplicity backups to a local NAS, and from that, duplicity to remote
storage.

Now, the CPU on the NAS isn't very powerful, and since the first round of
duplicity backup will already compress the files by default, I would like
one or both of the following to be implemented:

1) Add an option for tar to skip compression
or
2) Make a more general set of options to control the behaviour of tar, e.g.
setting compression level, setting compression type, omitting compression
etc.

One possibility would be to implement 1), which should not be too
difficult, and then work on 2) at a more leisurely pace.

Best regards,

Sune Mølgaard
```

Story points: 6.0
gpg-options bug (lp:#1367427)

[Original report](https://bugs.launchpad.net/bugs/1367427) created by **BRULE Herman (brule-herman)**
```
Hello,

--gpg-options "--compress-algo=bzip2 --bzip2-compress-level=9" with
duplicity do a bug:
GPGError: GPG Failed, see log below:
===== Begin GnuPG log =====
usage: gpg [options] [filename]
===== End GnuPG log =====

Gpg work with this options:
gpg --compress-algo=bzip2 --bzip2-compress-level=9 --output doc.txt.gpg
--symmetric doc.txt

Cheers,
```
Original tags: gpg

Story points: 12.0
Duplicity fails with "Invalid data - SHA1 hash mismatch for file", does not retry download.  (lp:#1360734)

[Original report](https://bugs.launchpad.net/bugs/1360734) created by **Eric O'Connor (oconnore)**
```
There appears to have been a corruption issue in my file download, about
halfway through an 80GB restore.

Invalid data - SHA1 hash mismatch for file:
 duplicity-full.20140720T050902Z.vol43.difftar.gpg
 Calculated hash: ee843c75152a61fd20c59e8cd3345288c0c52203
 Manifest hash: 5971136422f4646cb06d722cf188bb04b7df810f

When I checked the file hash on Amazon, it was exactly
5971136422f4646cb06d722cf188bb04b7df810f.

I would expect duplicity to retry the download when the hash doesn't match.

Duplicity version is: duplicity 0.6.24
Python: 2.7.8
Debian jessie/sid
```

Story points: 18.0
Duplicity fails during backup in with_tempdir() (lp:#1348193)

[Original report](https://bugs.launchpad.net/bugs/1348193) created by **shankao (shankao)**
```
Description:    Ubuntu Utopic Unicorn (development branch)
Release:        14.10

duplicity:
  Installed: 0.6.23-1ubuntu5
  Candidate: 0.6.23-1ubuntu5
  Version table:
 *** 0.6.23-1ubuntu5 0
        500 http://au.archive.ubuntu.com/ubuntu/ utopic/main amd64 Packages
        100 /var/lib/dpkg/status

During a full system backup, using the deja-dup, the process gets about
half-way through and then stops with the following error:
Backup Failed
Failed with an unknown error.

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1494, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1488, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1337, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1458, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 542, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 403, in write_multivol
    globals.gpg_profile, globals.volsize)
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 324, in
GPGWriteFile
    file = GPGFile(True, path.Path(filename), profile)
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 105, in
__init__
    self.logger_fp = tempfile.TemporaryFile( dir=tempdir.default().dir() )
  File "/usr/lib/python2.7/tempfile.py", line 497, in TemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags)
  File "/usr/lib/python2.7/tempfile.py", line 239, in _mkstemp_inner
    fd = _os.open(file, flags, 0600)
OSError: [Errno 2] No such file or directory: '/tmp/duplicity-
DEgmK7-tempdir/tmplcuF1s'

The referred to directory does not exist, however, the following one does
exist:
/tmp/duplicity-l4bYos-tempdir/
```

Story points: 26.0
librsync signature collisions/preimages (lp:#1342721)

[Original report](https://bugs.launchpad.net/bugs/1342721) created by **mik (therealmik)**
```
Note: this vulnerability is already public:
https://github.com/librsync/librsync/issues/5

Sending a copy to duplicity, because I found this when auditing duplicity.

The signatures generated by librsync for duplicity are md4 sums truncated
to 64 bits.

Collision attacks:
  An attacker could try to get two different entries into a signature file,
in case the first one to be seen will be the only one replicated in delta
files.
  This has almost no complexity for md4, and even for a strong hash
truncated to 8 bytes would only require approximately 2^32 hashes to be
generated (birthday collision).

Second-Preimage attacks:
  An attacker could compute a hash for existing (but known) data that they
want replicated elsewhere, and generate a different block that will later
be replaced with the original.

The main precondition for these attacks is that attacker-controlled data
needs to be written to a file then taken out of context after restore.
This is probably not an issue for a majority of users, but this could have
security ramifications for people backing up databases,  VM images or even
log files.

Replacing md4 in librsync with a 256-bit hash (blake2b is fast, sha256 is
standard) would be the best option, obviously requiring a change in the
signature file format (but not the delta file format - so backups could
still be restored using older versions).
```

Story points: 260.0
When incorrect ssh path provided coll-status always results in empty backup listing (lp:#1327792)

[Original report](https://bugs.launchpad.net/bugs/1327792) created by **Olivier Berger (oberger)**
```
Copying here a bug report already reported in Debian BTS, which I doubt is
specific to Debian :

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=750893

Let's pretend I'm doing a collection-status on an existing backup dir with
:
$ duplicity collection-status --ssh-backend=pexpect --use-scp --ssh-askpass
scp://root@aserver.local/mnt/backups/whatever

Then, it will create a mnt/backups/whatever inside rooot's home on
aserver.local which didn't exist already, instead of reporting an error
(the URl should be scp://root@aserver.local//mnt/backups/whatever for
/mnt/backups/whatever which does contain an existing backup).

collection-status shouldn't do any directory creation and only report
failure if the first dir of the path is missing.

Instead, as exhibited with -v9, it iterates on creating alll dirs of the
path, and then tries to list its contents, which most liely will  be empty.

Subsequent attempts by the user to debug will most likely too result in
trying other duplicity options instead of fixing the wrong path, as there's
nothing like 'dir mnt not found' like error.

Thanks in advance.
```

Story points: 6.0
gpg wait thread broken (on some pythons) (lp:#1326944)

[Original report](https://bugs.launchpad.net/bugs/1326944) created by **Andrew Stubbs (ams-codesourcery)**
```
When duplicity launches GPG it uses a thread to contain the waitpid call,
but this doesn't work on my machine. I have python 2.6 on an ARMv5te
system, and I'm guessing this is somehow significant.

The problem is, I think, that Python threading isn't true threading (in my
case), and the wait thread blocks the main thread. The result is that it
wait for GPG to exit before it has transmitted the passphrase or any of the
data. This means that the whole process hangs up, and no backup happens.

Anyway, whatever the precise cause, if I remove the whole thread, and
replace the "thread.join" call with a direct call to waitpid, then all
works as expected.

In fact, the current use of waitpid appears to show a misunderstanding of
how it works; for example, there's a debug log line that suggests it can
fail if waitpid is called after the child has already exited, which I don't
believe is true, in Linux. That said, perhaps there's an OS out there that
works differently?
```

Story points: 6.0
--include option with range in [] fails if first item of the range has no match (lp:#1326472)

[Original report](https://bugs.launchpad.net/bugs/1326472) created by **Andrea Ballarati (ballarati)**
```
duplicity 0.6.18
Python 2.7.3
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.4 LTS"
target filesystem Linux ext4

command line:
-v8 --ssh-askpass --volsize 2000 --no-encryption --asynchronous-upload
--log-file /var/log/backupscript.log --exclude **/temp/** --exclude **.tmp
--exclude **.bkp --include ignorecase:/srv/pub/rete/[q..z]** --exclude
/srv/pub/rete /srv/pub/rete
sftp://bkpusr@192.168.0.249//mnt/archivio/backup/

This command line fails (no files are backed up) since, I think, there is
no directory that begins with "q".
If I write instead /[qrstuwxyz]** backup works as expected (i.e. starts
from the directories that begins with "r").

The same command line works if I write /[r..z]**
```

Story points: 6.0
No support of long key_id or full fingerprint (lp:#1324819)

[Original report](https://bugs.launchpad.net/bugs/1324819) created by **4dro (kwadronaut)**
```
Tried using the full fingerprint as recommended by gnupg: 'The use of key
Ids is just a shortcut, for all automated processing the fingerprint should
be used. ' [1] But that failed, same for using the long keyid.

Error: Sign key should be an 8 character hex string, like 'AA0E73D2'.
Error: Received '60DEADBEEF29E89B' instead.

Relying on the short keyid is a bad idea, they can be easily spoofed [2].
Long keyids can also relatively easy collide [3] and they can cause serious
side effects. [4] I haven't tested what happens when there are colliding
fingerprints.

It'd be nice if we can make use of full fingerprints.

I have tried to look at where the looks of a key are tested but got a bit
lost.

[1] https://www.gnupg.org/documentation/manuals/gnupg-devel/Specify-a-User-
ID.html
[2] http://www.asheesh.org/note/debian/short-key-ids-are-bad-news
[3] http://thread.gmane.org/gmane.ietf.openpgp/7413
[4] https://www.debian-administration.org/users/dkg/weblog/105
```

Story points: 10.0
Wishlist: Ability to fund development (lp:#1322411)

[Original report](https://bugs.launchpad.net/bugs/1322411) created by **Ringo Kamens (ringokamens-deactivatedaccount)**
```
I looked on the Duplicity website but was unable to find any place to
donate money to fund development. Can we get this added to the
site/manpage/something? It would be great if a bug bounty program would be
opened, I use duplicity on one of my servers and would gladly contribute to
bugfixed and features.
```

Story points: 6.0
Giving up after 5 attempts. BackendException: Bad status code 200 reason OK. (lp:#1319557)

[Original report](https://bugs.launchpad.net/bugs/1319557) created by **L0RE (andreas-vogler)**
```
When i make a Backup With Duply (Duplicity). It works

Using installed duplicity version 0.6.21, python 2.7.5+, gpg 1.4.14 (Home:
~/.gnupg), awk 'mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan',
bash '4.2.45(1)-release (x86_64-pc-linux-gnu)'.
Autoset found secret key of first GPG_KEY entry 'XXXXXX' for signing.
Test - Encrypt to 88E30CC3 & Sign with XXXXX (OK)
Test - Decrypt (OK)
Test - Compare (OK)
Cleanup - Delete '/tmp/duply.17059.1400099642_*'(OK)

--- Start running command PRE at 22:34:03.210 ---
Skipping n/a script '/root/.duply/www/pre'.
--- Finished state OK at 22:34:03.223 - Runtime 00:00:00.012 ---

--- Start running command BKP at 22:34:03.234 ---
Using archive dir: /root/.cache/duplicity/duply_www
Using backup name: duply_www
Import of duplicity.backends.ftpsbackend Succeeded
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.u1backend Succeeded
Import of duplicity.backends.sshbackend Succeeded
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.ftpbackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Import of duplicity.backends.cloudfilesbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Using WebDAV protocol http
Using WebDAV host webdav.opendrive.com port None
Using WebDAV directory /web-server/
Reading globbing filelist /root/.duply/www/exclude
Main action: inc
================================================================================
duplicity 0.6.21 (January 23, 2013)
Args: /usr/bin/duplicity --name duply_NAME --encrypt-keyXXXX --sign-key
XXXX --verbosity 5 -v5 --exclude-globbing-filelist
/root/.duply/NAME/exclude / webdavs://USER@HOST/DIR
Linux SERVERNAME  3.11.0-20-generic #35-Ubuntu SMP Fri May 2 21:32:49 UTC
2014 x86_64 x86_64
/usr/bin/python 2.7.5+ (default, Feb 27 2014, 19:37:08)
[GCC 4.8.1]
================================================================================




UNTIL:

WebDAV PUT /web-server/duplicity-full.20140514T144621Z.vol83.difftar.gpg
request with headers: {'Connection': 'keep-alive', 'Authorization': 'Basic
STRING'}
WebDAV data length: 26271796
WebDAV response status 200 with reason 'OK'.
Attempt 1 failed. BackendException: Bad status code 200 reason OK.



This goes vor 5 TImes:

Giving up after 5 attempts. BackendException: Bad status code 200 reason
OK.
22:37:38.770 Task 'BKP' failed with exit code '50'.
--- Finished state FAILED 'code 50' at 22:37:38.770 - Runtime 00:03:35.535
---

--- Start running command POST at 22:37:38.787 ---
Skipping n/a script '/root/.duply/www/post'.
--- Finished state OK at 22:37:38.797 - Runtime 00:00:00.010 ---
```

Story points: 6.0