giseldo/neodataset huggingface dataset:
Support for enterprise OneDrive (lp:#1729689)

[Original report](https://bugs.launchpad.net/bugs/1729689) created by **Cysioland (cysioland)**
```
Currently only "mainstream" OneDrive is supported, there's no native way to
access sharepoint.com storage.

Some info that may be useful on tackling that:
https://micreabog.wordpress.com/2017/02/24/using-duplicity-with-microsoft-
sharepointonedrive-for-business/
```
Original tags: microsoft office365 onedrive sharepoint wishlist

Story points: 12.0
pyrax deprecated (lp:#1723894)

[Original report](https://bugs.launchpad.net/bugs/1723894) created by **Alexandre ZANNI (shark-oxi)**
```
Rackspace has deprecated pyrax (https://github.com/rackspace/pyrax/) in
favor of openstacksdk (https://pypi.python.org/pypi/openstacksdk) and
rackspacesdk (https://pypi.python.org/pypi/rackspacesdk).

ref: "python-cloudfiles deprecated"
https://bugs.launchpad.net/duplicity/+bug/1179322
```

Story points: 6.0
Backup from LVM snapshot fail without error stats / exit code (lp:#1722203)

[Original report](https://bugs.launchpad.net/bugs/1722203) created by **Bouke (bouke-haarsma)**
```
We're using duplicity to create backups of an LVM volume by creating a
snapshot. However we found that when a snapshot exceeds the allotted disk
space and the volume is forcefully unmounted, that duplicity handles this
rather unfortunately.

Actual results:
* Some errors will be logged on stdout
* Exit code is 0
* Backup statistics shows 0 errors

Expected results:
* Errors will be logged on stderr
* Exit code is != 0
* Backup statistics shows >0 errors

Our setup / problem:
* LVM disk with XFS file system
* LVM snapshot of that disk, mounted
* Duplicity backing up from the mounted snapshot
* LVM snapshot might get invalidated before backup finishes if there's a
lot of disk activity
* LVM will force unmount the snapshot once it is invalidated

Output:
```stdout
$ duplicity ...; echo $?
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Wed Sep 27 16:01:06 2017
Error [Errno 5] Input/output error getting delta for /XXX-
backup/data/local/collection-12--2763165517992512287.wt
Error [Errno 5] Input/output error getting delta for /XXX-
backup/data/local/collection-12--2763165517992512287.wt
--------------[ Backup Statistics ]--------------
StartTime 1507540632.42 (Mon Oct  9 11:17:12 2017)
EndTime 1507540788.59 (Mon Oct  9 11:19:48 2017)
ElapsedTime 156.17 (2 minutes 36.17 seconds)
SourceFiles 555
SourceFileSize 27970866720 (26.0 GB)
NewFiles 35
NewFileSize 14705246318 (13.7 GB)
DeletedFiles 0
ChangedFiles 45
ChangedFileSize 11136098320 (10.4 GB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 80
RawDeltaSize 5603945441 (5.22 GB)
TotalDestinationSizeChange 2668791306 (2.49 GB)
Errors 0
-------------------------------------------------
0
```

Steps to reproduce:
```
$ lvcreate -l100M --name backup --snapshot VG/LV
$ mount -o nouuid VG/backup /backup
$ duplicity /backup ... &
$ dd if=/dev/zero of=/backup/zeroes bs=1M count=101
```

Duplicity: 0.7.13.1
Python: 2.7.5
OS: CentOS Linux release 7.3.1611 (Core)
Filesystem: XFS / LVM snapshot
```

Story points: 6.0
I'm not sure if this is a error or my script mistake (lp:#1721535)

[Original report](https://bugs.launchpad.net/bugs/1721535) created by **Bruno (brunooliveira1)**
```
I try to make a backup using command syntax
/usr/local/bin/duplicity full-if-older-than 1M / --encrypt-key=gpgkey
--asynchronous-upload file:///backup
Os metadados remotos e locais estão sincronizados; nenhuma sincronização é
necessária.
Data da última cópia de segurança completa: Thu Sep 14 21:05:58 2017
Traceback (most recent call last):
  File "/usr/local/bin/duplicity", line 1548, in <module>
    with_tempdir(main)
  File "/usr/local/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/local/bin/duplicity", line 1383, in main
    do_backup(action)
  File "/usr/local/bin/duplicity", line 1515, in do_backup
    check_last_manifest(col_stats)  # not needed for full backup
  File "/usr/local/bin/duplicity", line 1217, in check_last_manifest
    last_backup_set.check_manifests()
  File "/usr/local/lib/python2.7/dist-packages/duplicity/collections.py",
line 208, in check_manifests
    remote_manifest = self.get_remote_manifest()
  File "/usr/local/lib/python2.7/dist-packages/duplicity/collections.py",
line 244, in get_remote_manifest
    (self.remote_manifest_name, str(message)))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfa' in
position 169: ordinal not in range(128)
```
Original tags: backup incremental

Story points: 6.0
duplicity 0.7.14 occasionally looping when backup up to swift object store (lp:#1715280)

[Original report](https://bugs.launchpad.net/bugs/1715280) created by **Doug (caniwi)**
```
Running duplicity 0.7.14 on Ubuntu 14.04 with Python 2.7.6.  From time to
time, duplicity goes into a loop.  This behaviour also occurred on
duplicity 0.7.13.

An strace on the task shows the following:
dev-sugar-combined00:~# ps aux |grep dup
root     15328  0.0  0.0  11760  2072 pts/1    S+   14:05   0:00 grep dup
root     24704  0.0  0.0   4460   796 ?        Ss   Sep01   0:00 /bin/sh -c
/usr/local/bin/duplicity_backup.sh chensi7_mysql_sgr_chensi7 >>
/var/log/backup/chensi7_mysql_sgr_chensi7.log 2>&1
root     24708  0.0  0.0  12476  2892 ?        S    Sep01   0:00 /bin/bash
/usr/local/bin/duplicity_backup.sh chensi7_mysql_sgr_chensi7
root     24836  0.3  0.3 223624 30632 ?        S    Sep01  28:02
/usr/bin/python /usr/bin/duplicity --verbosity Notice --full-if-older-than
7D --num-retries 3 --asynchronous-upload --no-encryption --volsize 512
/var/opt/backups/sgr_chensi7 swift://dev-sugar-combined00.cloudtech.co.nz-
chensi7_mysql_sgr_chensi7
dev-sugar-combined00:~# strace -p 24836
Process 24836 attached
select(0, NULL, NULL, NULL, {0, 23735}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 36067}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 1000})  = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 2000})  = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 4000})  = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 8000})  = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 16000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 32000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 35575}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 1000})  = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 2000})  = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 4000})  = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 8000})  = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 16000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 32000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 35583}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 1000})  = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 2000})  = 0 (Timeout)
```

Story points: 6.0
Boto backend makes wrong assumption about Glacier class (lp:#1713369)

[Original report](https://bugs.launchpad.net/bugs/1713369) created by **Martin (martin3000)**
```
The code in _boto_single.py in function pre_process_download assumes that
an S3 object must not be in storage class "Glacier" to be downloaded. This
is not correct. S3 objects in Glacier can be downloaded after a restore has
been triggered and when the retoration-expire-date is available. This means
that there is a temporary copy which and be downloaded.

Furthermore, pre_process_download is called first from sync_archive and
later from copy_to_local, so it must be avoided to trigger the restore two
times.

My version looks like this:

    def pre_process_download(self, remote_filename, wait=False):
        # Used primarily to restore files in Glacier
        key_name = self.key_prefix + remote_filename
        if not self._listed_keys.get(key_name, False):
            self._listed_keys[key_name] =
list(self.bucket.list(key_name))[0]
        key = self._listed_keys[key_name]
        key2 = self.bucket.get_key(key.key)

        if key2.storage_class == "GLACIER":
            if not key2.expiry_date: # no temp copy avail
                if not key2.ongoing_restore:
                    log.Info("File %s is in Glacier storage, restoring" %
remote_filename)
                    key.restore(days=2)  # we need the copy for 2 days
                if wait:
                    log.Info("Waiting for file %s to restore in Glacier" %
remote_filename)
                    while not self.bucket.get_key(key.key).expiry_date:
                       time.sleep(60)
                       self.resetConnection()
                    log.Info("File %s was successfully restored in Glacier"
% remote_filename)
```
Original tags: backend boto glacier restore

Story points: 6.0
Calling pre_process_download does not work (lp:#1713367)

[Original report](https://bugs.launchpad.net/bugs/1713367) created by **Martin (martin3000)**
```
Duplicity Version 0.7.06 on Linux

In function sync_archive of duplicity at line 1185 I find the following
code:

            if hasattr(globals.backend, 'pre_process_download'):
                globals.backend.pre_process_download(local_missing)
            for fn in local_missing:
                copy_to_local(fn)

Before copy_to_local, some preprocessing should be made. In case of Amazon
S3 (boto) this should restore objects from Glacier. But "globals.backend"
point to the backend wrapper so the condition if hasattr(globals.backend,
'pre_process_download') is never true. It must be globals.backend.backend.

When calling pre_process_download the argument is "local_missing" which is
a list. But _boto_single.py expects a single filename as the argument for
pre_process_download.

I would change this to:

            if hasattr(globals.backend.backend, 'pre_process_download'):
                for fn in local_missing:
globals.backend.backend.pre_process_download(fn,wait=False) # restore from
S3
            for fn in local_missing:
                copy_to_local(fn)

Same issue in function restore_get_patched_rop_iter.
```
Original tags: boto glacier s3

Story points: 6.0
"<target_folder>" for restore action is new name for restored file, and not the target folder (lp:#1708501)

[Original report](https://bugs.launchpad.net/bugs/1708501) created by **ardabro (ardabro)**
```
v: 7.11

> mkdir restore
> ls -l
total 0
drwxr-xr-x 1 user users 0 Aug  3 19:23 restore
>
> duplicity restore --file-to-restore="user/file.txt"
file:///mnt/backup/duplicity ./restore/
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Thu Aug  3 00:28:09 2017
GnuPG passphrase for decryption:
>
> ls -l
total 32
-rw-r--r-- 1 user users 20749 Jan 13  2016 restore

It deletes target folder ("restore") and renames extracted file to its
name!
There is no way to extract specific file with its original name to
arbitrary location.
```

Story points: 6.0
Restore fails if backup contains filenames with invalid utf-8 encodings (lp:#1707461)

[Original report](https://bugs.launchpad.net/bugs/1707461) created by **Bernie Innocenti (codewiz)**
```
This is duplicity 0.7.13.1 trying to restore a backup created by the same
version:

Error '(u'Error creating directory /run/media/bernie/My
Book/restore/public_html/projects/amiga/XModule/XModuleSrc/Catalogs/Fran\ufffdais',
7)' processing
public_html/projects/amiga/XModule/XModuleSrc/Catalogs/Fran�ais
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1540, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1534, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1385, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1462, in do_backup
    restore(col_stats)
  File "/usr/bin/duplicity", line 728, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/usr/lib64/python2.7/site-packages/duplicity/patchdir.py", line
560, in Write_ROPaths
    ITR(ropath.index, ropath)
  File "/usr/lib64/python2.7/site-packages/duplicity/lazy.py", line 344, in
__call__
    last_branch.fast_process, args)
  File "/usr/lib64/python2.7/site-packages/duplicity/robust.py", line 38,
in check_common_error
    return function(*args)
  File "/usr/lib64/python2.7/site-packages/duplicity/patchdir.py", line
614, in fast_process
    ropath.copy(self.base_path.new_index(index))
  File "/usr/lib64/python2.7/site-packages/duplicity/path.py", line 445, in
copy
    other.writefileobj(self.open("rb"))
  File "/usr/lib64/python2.7/site-packages/duplicity/path.py", line 622, in
writefileobj
    fout = self.open("wb")
  File "/usr/lib64/python2.7/site-packages/duplicity/path.py", line 564, in
open
    result = open(self.name, mode)
IOError: [Errno 84] Invalid or incomplete multibyte or wide character:
'/run/media/bernie/My
Book/restore/public_html/projects/amiga/XModule/XModuleSrc/Catalogs/fran\xe7ais.ct'

The file is named "français.ct" but using the latin-1 encoding of "ç".
```

Story points: 28.0
Allow per-directory include-filelist (lp:#1705934)

[Original report](https://bugs.launchpad.net/bugs/1705934) created by **Chris Hunt (chrahunt)**
```
Currently to include/exclude paths we must provide all exclusions to
duplicity up front, when executing the command. This does not compose well,
as individual directories within a source directory may have different and
even conflicting requirements on what contained files should be backed up.
It would be very convenient if we could specify a filename, then if
duplicity sees a file with that name in a directory it is backing up, it
will read it as if it had been passed to --include-filelist, with the
exceptions:

1. Selection rules are interpreted relative to the directory of the file in
which they were found
2. Selection rules apply only in the directory in which the matching file
was found and child directories (recursively)
3. Selection rules are given priority the lower their originating file is
in the directory hierarchy (assuming the root is at the top). Any selection
rules provided on the command line or via --include-filelist would be
applied last

This would behave similar to .gitignore in git, and has similar benefits:

1. Keep information about relevant files to include/exclude closer to the
impacted files
2. Move directories with special child file selection rules without
requiring updating a global file with the new path
3. Eliminate the need for external tools to generate selection rules, and
reduce extreme uses of the global selection rule list (e.g. 1576389)
4. More optimal behavior w.r.t. file selection since the selection rules
that are most likely to apply to a file would be tested first
```
Original tags: wishlist

Story points: 6.0
duplicity fails to restore backup with key error: 1 on volume_name_dict (lp:#1705849)

[Original report](https://bugs.launchpad.net/bugs/1705849) created by **Hobson Lane (hobs)**
```
I upgraded a laptop from 16.04 to 17.04 and could not resolve the perpetual
login prompt boot so I deleted my ubuntu partition reinstalled ubuntu 17.04
and did a boot recovery to restore the grub prompt for dual boot. However,
when I attempted to restore my user data from a deja-dupe backup (from
16.04) I received the following python 2.7.13 Traceback error:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1532, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1526, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1380, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1457, in do_backup
    restore(col_stats)
  File "/usr/bin/duplicity", line 722, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py",
 560, in Write_ROPaths
    for ropath in rop_iter:
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 523,
in integrate_patch_iters
    for patch_seq in collated:
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 389,
in yield_tuples
    setrorps(overflow, elems)
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 378,
in setrorps
    elems[i] = iter_list[i].next()
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 121,
in difftar2path_iter
    tarinfo_list = [tar_iter.next()]
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 339,
in next
    self.set_tarfile()
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 333,
in set_tarfile
    self.current_fp = self.fileobj_iter.next()
  File "/usr/bin/duplicity", line 758, in get_fileobj_iter
    backup_set.volume_name_dict[vol_num],
KeyError: 1

Environment:

Python 2.7.13 (default, Jan 19 2017, 14:48:08)
[GCC 6.3.0 20170118] on linux2

duplicity 0.7.06

deja-dup 34.4
```

Story points: 6.0
Only sync relevant remote metadata to prevent big cache (lp:#1702715)

[Original report](https://bugs.launchpad.net/bugs/1702715) created by **Markus (doits)**
```
Duplicity version: 0.7.13.1
Python version: 2.7.12
OS Distro and version: Ubuntu 16.04.2
Type of target filesystem: Multiple

Hello,

this is a follow up of http://lists.nongnu.org/archive/html/duplicity-
talk/2017-07/msg00000.html (slightly modified).

I've been using duplicity to backup since 2014 with a full backup every 3
months, so I have some secondary backup chains. Additionally backups go to
multiple separate places (each having multiple backup chains). My local
cache folder is now over 70gb big and I had the idea to delete the cache
for the secondary chains to free up space - since they are only needed when
restoring files, but not for new backups (to the primary chains).

So I deleted the cache files (for example duplicity-full-
signatures.20141106T121637Z.sigtar.gpg) and reclaimed a lot of space.

But after the next backup run, I saw this in the logs:

Synchronizing remote metadata to local cache...
Copying duplicity-full-signatures.20141106T121637Z.sigtar.gpg to local
cache.
Copying duplicity-full-signatures.20150505T234205Z.sigtar.gpg to local
cache.
Copying duplicity-full-signatures.20150904T234123Z.sigtar.gpg to local
cache.
Copying duplicity-full-signatures.20151205T004128Z.sigtar.gpg to local
cache.
...

My current workaround is: Before I create a new full backup, I move the
remote backup files to a new location. So for duplicity it looks like the
backup location is fresh and it even deletes the local cache automatically,
starting with a full backup (which I wanted anyways).

If I later want to restore from that backup I will have to move the files
back into place on that remote.

It would be great to have a command line option to prevent syncing all
metadata but the current one. Maybe it could even determine which metadata
is needed for the current operation and only sync this one, so for example
if I wanted to restore from a secondary backup chain it would only sync the
secondary backup chain's metadata. So if metadata was deleted manually but
not needed for the current operation, it would stay deleted.
```
Original tags: cache metadata

Story points: 16.0
Ignore Errors on restore (lp:#1702696)

[Original report](https://bugs.launchpad.net/bugs/1702696) created by **Robert Coup (rcoup)**
```
Situation:

1. full backup with good signatures/manifest but one or more corrupted
volumes (eg. `echo "horse" > myarchive/duplicity-
full.20170706T134837Z.vol9.difftar.gpg`)

2. Error messages along the lines of:

        Invalid data - SHA1 hash mismatch for file:
         duplicity-full.20170706T134837Z.vol9.difftar.gpg
         Calculated hash: 7b64f0f207214f9894a2f4d08a95e57f3c773e72
         Manifest hash: 90f449bb4662db4242caab58f509ca5354afb631

3. `--ignore-errors` doesn't ignore these

The error comes from the de-GPG'ing code in restore_get_enc_fileobj(). We
can ignore the errored volume (obviously ignoring the data in it) and
continue restoring what we can by proceeding to the next volume.

Proof-of-concept patch is attached.

* Doesn't actually print the name of the missing/bad/etc files, just the
volumes. The index we're restoring is stored a few levels further up the
stack.
* Does work for files which span multiple volumes including corrupted, but
the specific files will be hosed/bad.
* Doesn't work if directories don't exist (eg. corruption is in volume 1)
but later volumes want to write to files in them. Presumably with --force
and pre-creating you can workaround
* No idea what happens if GPG is disabled or the file is missing, but
presumably similar to handle.
* No idea what happens with incrementals either :)

Related:
* https://bugs.launchpad.net/deja-dup/+bug/487720
```
Original tags: restore

Story points: 10.0
Support Sia backup storage (lp:#1700005)

[Original report](https://bugs.launchpad.net/bugs/1700005) created by **Wernight (werner-beroux)**
```
*Sia* (http://sia.tech/get-started/) is relatively easy to setup for users
but it seems to really shine for backup storage for now.

The UI flow is shown at https://blog.sia.tech/getting-started-with-private-
decentralized-cloud-storage-c9565dc8c854, but there is *Sia Daemon*
 (http://sia.tech/apps/) (see API at https://blog.sia.tech/api-quickstart-
guide-f1d160c05235).

Why Sia? One arguable reason is that it's a distributed system with copies
on 3 machines. Yes it's very new and linked to a crypto currency, but the
real No.1 reason is the cost: About $0.16 / TB / month
(https://www.reddit.com/r/siacoin/comments/5vl5an/sia_storage_costs_016_per_tb_per_month/).

For a comparison:
 - $10 /TB/mo Google Cloud Nearline storage
 - $7 /TB/mo Google Cloud Cold storage
 - $12 /TB/mo S3 Infrequent storage
 - $4 /TB/mo S3 Galcier storage
```

Story points: 16.0
Does not preserve timestamp of symbolic links (lp:#1697157)

[Original report](https://bugs.launchpad.net/bugs/1697157) created by **Phil Ruffwind (rufflewind)**
```
It seems that duplicity does not preserve the timestamp (specifically
modification time) of symbolic links.

Test case:

1. Create some symbolic links in a directory A (possibly broken links,
doesn't matter).  Wait a minute or so.
2. duplicity A file://B
3. duplicity file://B C

Expected: same timestamp shown with ls -l as in directory A
Actual: timestamp of symbolic links is the time of restore (step 3) instead

Does not affect normal files, just symbolic links.

duplicity 0.7.12
Python 3.6.1
Linux 4.11.3-1-ARCH
```

Story points: 6.0
ftpes:// not docummented (lp:#1696977)

[Original report](https://bugs.launchpad.net/bugs/1696977) created by **Yajo (yajo)**
```
I was becoming crazy setting up an ftpes:// backup, and the main problem
was that I was using ftps:// instead.

duplicity man page says nothing about ftpes support, nor does the website.
I had to guess that by https://bugs.launchpad.net/duplicity/+bug/1072130.

Would be great if you add it, so next one does not become crazy :)

FTR: differences: https://www.cerberusftp.com/support/help/ftp-support/
```

Story points: 6.0
Amazon Drive backend fails after 1 hour (lp:#1689501)

[Original report](https://bugs.launchpad.net/bugs/1689501) created by **Rojetto (rojetto)**
```
I'm using duplicity 0.7.12 with the Amazon Drive backend adbackend.py
(revision 1218) from series 0.8 installed.

When backing up a larger amount of data, the first few volumes upload fine,
but the upload for later volumes starts to fail after exactly 1 hour with
'HTTP Error: 401 Client Error: Unauthorized'. I don't know how this stuff
exactly works, but it seems to me like the backend script requests an OAuth
token just at the beginning of the process, that simply expires after a
while. Maybe refreshing the token before every volume upload would be an
option?
```
Original tags: amazon authorization backend drive token unauthorized

Story points: 6.0