Repository: borgbackup/borg

Issue:
Disk space is calculated wrongly on macOS (and possibly elsewhere)
<!--
Thank you for reporting an issue.

*IMPORTANT* -  *before* creating a new issue please look around:
 - Borgbackup documentation: http://borgbackup.readthedocs.io/en/stable/index.html
 - FAQ: https://borgbackup.readthedocs.io/en/stable/faq.html
 and
 - open issues in Github tracker: https://github.com/borgbackup/borg/issues
  
If you cannot find a similar problem, then create a new issue.

Please fill in as much of the template as possible.
-->

## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

Bug

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).
`1.2.0.dev`

#### Operating system (distribution) and version.
macOS 10.14.3

#### Hardware / network configuration, and filesystems used.
APFS file system

#### How much data is handled by borg?
less than 1 GB

#### Full borg commandline that lead to the problem (leave away excludes and passwords)
`$ borg create --debug ~/Downloads/test-repo::jdkdjfdd ~/Pictures/wallpapers`

## Describe the problem you're observing.

I noticed that the free space calculation, as [done by borg](https://github.com/borgbackup/borg/blob/2bcff382cbe8d390cf526003697dfa2cf6b4cf47/src/borg/repository.py#L661) doesn't make sense on macOS:

```
In [7]: st_vfs = os.statvfs('/')

In [8]: st_vfs.f_bavail * st_vfs.f_frsize
Out[8]: 50595008512

In [9]: st_vfs.f_bavail * st_vfs.f_bsize
Out[9]: 12952322179072

In [10]: os.statvfs('/')
Out[10]: os.statvfs_result(f_bsize=1048576, f_frsize=4096, f_blocks=61202533, f_bfree=13309382, f_bavail=12351802, f_files=4294967295, f_ffree=4292317517, f_favail=4292317517, f_flag=0, f_namemax=255)
```

`f_bsize` is too high and results in ~13 TB of free space. (real value: ~50 GB). Apple's [docs](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/statvfs.3.html) say that `f_bsize` is "The preferred length of I/O requests for files on this file system.", which doesn't sound like the block size.

`f_frsize` may be the right variable. This is also mentioned in this older [SO issue](https://stackoverflow.com/questions/787776/find-free-disk-space-in-python-on-os-x). For Borg it may be best to just use the higher-level `shutil.disk_usage`. It seems to give the result we need: (same system as above)

```
In [13]: shutil.disk_usage('/')
Out[13]: usage(total=250685575168, used=196174888960, free=50588438528)
```

#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.
Yes

#### Include any warning/errors/backtraces from the system logs
`$ borg create --debug ~/Downloads/test-repo::jdkdjfdd ~/Pictures/wallpapers`
```
using builtin fallback logging configuration
37 self tests completed in 0.14 seconds
Verified integrity of /Users/manu/Downloads/test-repo/index.137
Enter passphrase for key /Users/manu/Downloads/test-repo:
TAM-verified manifest
security: read previous location '/Users/manu/Downloads/test-repo'
security: read manifest timestamp '2019-01-31T03:55:02.282390'
security: determined newest manifest timestamp as 2019-01-31T03:55:02.282390
security: repository checks ok, allowing access
Creating archive at "/Users/manu/Downloads/test-repo::jdkdjfdd"
Verified integrity of /Users/manu/.cache/borg/cc8bc42b3ae0b89e9d90453afa48a489601a5dc3c5769089ac7c6f22140b7593/chunks
Reading files cache ...
Verified integrity of /Users/manu/.cache/borg/cc8bc42b3ae0b89e9d90453afa48a489601a5dc3c5769089ac7c6f22140b7593/files
security: read previous location '/Users/manu/Downloads/test-repo'
security: read manifest timestamp '2019-01-31T03:55:02.282390'
security: determined newest manifest timestamp as 2019-01-31T03:55:02.282390
security: repository checks ok, allowing access
Processing files ...
Cleaned up 0 uncommitted segment files (== everything after segment 137).
Verified integrity of /Users/manu/Downloads/test-repo/hints.137
check_free_space: few segments, not requiring a full free segment
check_free_space: calculated working space for compact as 59173757 bytes
check_free_space: required bytes -1999940656985, free bytes 13431432282112
```

Last line gives the wrong free disk space.


Discussion:
yes, it needs to use frsize or the shutil stuff. needs fix in master and 1.1-maint.

maybe for now we rather want frsize as it is the minimal fix. from my experience, the stuff in shutil wasn't great sometimes - so better take no risk for 1.1-maint and do the minimal fix and not use something completely different. for now, i'ld do the same change in master to have both in sync.

if there is a need for more platform compatibility later, we can switch to shutl (not in 1.1-maint).

for our excuse: the (linux) man pages about statvfs f_bsize were not as clear as on macOS. :-|

from the freebsd man page:
```
	   f_frsize   The size in bytes	of the minimum unit of allocation on
		      this file	system.	 (This corresponds to the f_bsize mem-
		      ber of struct statfs.)
          f_bsize    The preferred length of I/O requests for files on	this
		      file system.  (Corresponds to the	f_iosize member	of
		      struct statfs.)
```
It totally helps to name the same stuff differently and different stuff the same (statvfs vs. statfs). :-(

Resolving patch:
diff --git a/src/borg/remote.py b/src/borg/remote.py
--- a/src/borg/remote.py
+++ b/src/borg/remote.py
@@ -1104,7 +1104,7 @@ def __init__(self, repository, pack=None, unpack=None, transform=None):
 
     def query_size_limit(self):
         stat_fs = os.statvfs(self.basedir)
-        available_space = stat_fs.f_bsize * stat_fs.f_bavail
+        available_space = stat_fs.f_bavail * stat_fs.f_frsize
         self.size_limit = int(min(available_space * 0.25, 2**31))
 
     def key_filename(self, key):
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -658,7 +658,7 @@ def check_free_space(self):
             logger.warning('Failed to check free space before committing: no statvfs method available')
             return
         # f_bavail: even as root - don't touch the Federal Block Reserve!
-        free_space = st_vfs.f_bavail * st_vfs.f_bsize
+        free_space = st_vfs.f_bavail * st_vfs.f_frsize
         logger.debug('check_free_space: required bytes {}, free bytes {}'.format(required_free_space, free_space))
         if free_space < required_free_space:
             if self.created:


Test patch: