Repository: learningequality/kolibri

Issue:
Free Space Report Is Wrong When Content is Symbolically Linked 
### Observed behavior

> $ ls -l /opt/kolibri/
total 219344
lrwxrwxrwx  1 root    root           44 Sep  7 21:19 content -> /media/esata/MEDIA_DISK/KOLIBRI_DATA/content
-rw-r--r--  1 kolibri kolibri 220577792 Sep  9 10:15 db.sqlite3
-rw-r--r--  1 kolibri kolibri     32768 Sep  9 10:41 db.sqlite3-shm
-rw-r--r--  1 kolibri kolibri     78312 Sep  9 10:41 db.sqlite3-wal
-rw-r--r--  1 kolibri kolibri         0 Sep  7 19:06 debug.log
-rw-r--r--  1 kolibri kolibri      4096 Sep  7 19:06 job_storage.sqlite3
-rw-r--r--  1 kolibri kolibri     32768 Sep  9 10:15 job_storage.sqlite3-shm
-rw-r--r--  1 kolibri kolibri   3761592 Sep  8 12:42 job_storage.sqlite3-wal
-rw-r--r--  1 kolibri kolibri     66774 Sep  9 10:15 kolibri.log
-rw-r--r--  1 kolibri kolibri       479 Sep  7 19:06 kolibri_settings.json
drwxr-xr-x  5 kolibri kolibri      4096 Sep  7 19:06 kolibri-venv
-rw-r--r--  1 kolibri kolibri      6538 Sep  9 10:41 server.log
-rw-r--r--  1 kolibri kolibri        10 Sep  9 10:15 server.pid
drwxr-xr-x  2 kolibri kolibri      4096 Sep  9 11:01 sessions
drwxr-xr-x 25 kolibri kolibri      4096 Sep  7 19:06 static

[me@tower Raccoon]$ df -h / /media/esata/MEDIA_DISK
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        39G   34G  2.4G  94% /
/dev/sdb2       1.8T  1.6T  140G  92% /media/esata/MEDIA_DISK

Note the symlinked location has 140G available but Kolibri front end reports "Your remaining space: 2 GB" 
…

### Expected behavior
Correct free space should be reported.

…

### User-facing consequences
Cannot install content.

…

### Errors and logs

Vivaldi browser 

1.15.1147.42 (Stable channel) (64-bit)
--

Kolibri version | 0.10.2
-- | --
http://192.168.1.100:8087/
/opt/kolibri/db.sqlite3
tower
Linux-4.14.65-1-MANJARO-x86_64-with-arch-Manjaro-Linux
2 GB




[kolibri.log.txt](https://github.com/learningequality/kolibri/files/2363937/kolibri.log.txt)
[server.log.txt](https://github.com/learningequality/kolibri/files/2363938/server.log.txt)



Discussion:
related: #3883
@lyw07 would you mind taking a look at this?
Yes sure. I will look into this issue 
Hi @nobicycle ,

Instead of creating symbolic links to your new disk to extend the storage space, I would recommend using our command `kolibri manage content movedirectory <destination>` to move your current content into the new disk and set the new disk folder to be the directory to store all the content. Here are the instructions for using the command: https://kolibri.readthedocs.io/en/latest/manage.html#change-the-location-of-kolibri-content-files

But even with using the command, the free space report is still wrong, as you pointed out in this issue. I would fix the issue under the circumstance that the directory of content files is reset through the command `kolibri manage content movedirectory <destination>`. 

Please let me know if you have any questions. Thank you for filing the issue!
@lyw07 

The link 'ln -s' command is the simplest way to achieve kolibri content on an alternate volume. It is completely transparent to the application and guaranteed not to cause issues. 

This method is referenced in your own Kolibri administration area of the manual. 

Performing some complicated trick in Python is clearly an inferior choice .

Most servers are now running Linux, even on the Microsoft Azure platform. 
Therefore this platform is the one you should be prioritizing and therefore you should be using its low-level application transparent solutions.

Even after running your "Python trick" the disk space is still reported wrong.
 
Hi @nobicycle -

The guidance that @lyw07 gave you is correct. It is not a "python trick", but a new Kolibri feature which allows the application to explicitly reference the content in a location outside of the Kolibri home folder.

> This method is referenced in your own Kolibri administration area of the manual.

Thank you for bringing this to our attention. We'll update the docs to remove this outdated advice.

It is important to use the `content movedirectory` command because it ensures that your database is kept annotated with all available content.

> Even after running your "Python trick" the disk space is still reported wrong.

As was clearly communicated, "even with using the command, the free space report is still wrong, as you pointed out in this issue".

The fix is referenced directly above and will be available in the 0.11 release.

> Performing some complicated trick in Python is clearly an inferior choice.

In the future, please be more respectful in the way that you express your opinions in our repos. For reference see our [code of conduct](https://kolibri-dev.readthedocs.io/en/develop/start/contributing/code_of_conduct.html).

Hi @indirectlylit and everyone else - the Code of Conduct team is there to resolve any grievances, disputes etc. If you feel that unacceptable behavior has taken place, just flag the team that handles this: codeofconduct@learningequality.org

-----

As a Linux user, I feel that making symbolic links is in my DNA. It's a very intuitive way of managing disk space in a flexible way, free from hierarchic directory structure. So I'd be very sorry if suddenly we didn't support it, and have to tell people who are using symbolic links are wrong.

The freedom to manage the file system of Kolibri is extremely critical, as it will contain large amounts of data. Users have to be able to schedule and manage disk space according to their tools, knowledge and needs.

Using symbolic links, mount points etc. are all part of this. Using `kolibri manage content movedirectory` is another alternative that's especially critical to ensure that inexperienced users have a safe method to move data. IMO they are not interchangeable.

Thank you @benjaoming for flagging this.

My apologies, @nobicycle: it was not my intent to accuse you of a violation. 

I really do appreciate your engagement and you've been a valuable member of our small development community over the past year, with many insightful contributions.


Resolving patch:
diff --git a/kolibri/utils/system.py b/kolibri/utils/system.py
--- a/kolibri/utils/system.py
+++ b/kolibri/utils/system.py
@@ -153,7 +153,7 @@ def get_free_space(path=KOLIBRI_HOME):
             raise ctypes.winError()
         result = free.value
     else:
-        st = os.statvfs(path)
+        st = os.statvfs(os.path.realpath(path))
         result = st.f_bavail * st.f_frsize
 
     return result


Test patch:
diff --git a/kolibri/core/device/test/test_api.py b/kolibri/core/device/test/test_api.py
--- a/kolibri/core/device/test/test_api.py
+++ b/kolibri/core/device/test/test_api.py
@@ -1,3 +1,4 @@
+import os
 import platform
 import sys
 from collections import namedtuple
@@ -199,7 +200,7 @@ def test_posix_freespace(self):
 
                 response = self.client.get(reverse("kolibri:core:freespace"), {'path': 'test'})
 
-                os_statvfs_mock.assert_called_with('test')
+                os_statvfs_mock.assert_called_with(os.path.realpath('test'))
                 self.assertEqual(response.data, {'freespace': 2})
 
     def test_win_freespace_fail(self):