Repository: bookwyrm-social/bookwyrm

Issue:
data export
you should be able to download your bookshelves and review data


Discussion:
I was wondering if there is a standard and found many based on a [quick search about library exchange format](https://www.minitex.umn.edu/services/digital-initiatives/standards-digital-collections/data-format-exchange-standards).

Do you already know which one you wanted to implement?
I think something like MARC wouldn't be a good fit here - I'm envisioning just a csv, similar to what you might download from goodreads or librarything. The data being downloaded is mostly user data, rather than library metadata.
I'm wondering if it would be ok for me to work on this since I heavily need a CSV export to have backups stored just in case. 
Yeah! Let me know if you have any questions 😄 
I'm waiting for a laptop I bought to arrive to work on this task. Apart from it being a CSV that is compatible with GoodReads. Do you have any set requirements for this particular task? Code style guidelines? In my case the requirements I always have is to check for accessibility, make sure if there is a way to test it to add that, and test for edge cases.
I think goodreads compatibility is a good baseline; it shouldn't be too hard to add columns if there's something people would find useful in the future. 

It may make sense to make the generation of the csv file happen in a celery task to avoid backing up the main site process - I'm just basing that on how other services serve CSVs.

I went ahead and wrote a basic code style thing for the documentation rather than stream my conscious out here - let me know if there's info that would be helpful to add there: https://github.com/bookwyrm-social/documentation/pull/17
Good to know. I will research the celery task when I get setup with that new computer that Dell refuses to send. If push comes to shove I'll set a VM for this task because I like the app a lot and use it. 
Since it all runs in docker, getting celery running _should_ be a non-issue (if you're using docker). But let me know if you run into any weirdness!
I finally got a laptop and I haven't set up a service with Nginx in a very long time. At least not locally so trying to figure that out in the meantime. Since it doesn't specify directories or how exactly to set it up without having done it before on the same app or elsewhere I'm going to poke at it for a bit. 
Quick update I think I'm doing good. The docker compose is doing most of the work. Just that initial part was a bit confusing without seeing the codebase. 
I'm running into some issues when running this command `docker-compose run --rm web python manage.py migrate`. 

```
Creating bookwyrm_web_run ... done
python: can't open file '/app/manage.py': [Errno 13] Permission denied
ERROR: 2
```

One thing that might be another issue is that the Dockerfile has this last command `RUN apt-get update && apt-get install -y gettext libgettextpo-dev && apt-get clean` and I'm running Fedora 34. Not Ubuntu. Let me know if there is something I'm missing on the docker-compose or if we can connect to get this initial setup. I will try and repay that back with some UI enhancements.
The `apt-get` commands are running in the docker container, so it shouldn't matter what operating system you're running (I'm on a mac, for example, which doesn't support `apt-get`), just what the docker image is installing. 

I wonder if you're hitting a system permissions problem with docker? This isn't a long term solution, but does it run with sudo?

This is what I found looking around briefly:
https://stackoverflow.com/a/60818019
https://docs.docker.com/engine/install/linux-postinstall/
https://linuxconfig.org/how-to-install-docker-on-fedora-linux-system#h2-run-docker-without-root

My theory is that the docker user might not have the permissions to do the action. I have been running it with sudo, but still run into the issue. I haven't found a way to test this before adding any permission that might be problematic. Will be taking a look at those resources later in the day, cause right now I'm at work. 
Cannot help specifically with the issue, however I see you mentioned backups. 
When implemented could anything be done such as a script ran on cron to automatically download backup copies of user profiles periodically? This may need an api token, which I don't think bookwyrm has implemented if I am correct?
This is something I'd be interested in, because instances can go down or die anytime, it would be nice to not just lose everything if it happens. Backups are good.
> Cannot help specifically with the issue, however I see you mentioned backups.
> When implemented could anything be done such as a script ran on cron to automatically download backup copies of user profiles periodically? This may need an api token, which I don't think bookwyrm has implemented if I am correct?
> This is something I'd be interested in, because instances can go down or die anytime, it would be nice to not just lose everything if it happens. Backups are good.

I think this is off topic for this issue, it would be closely related to #1012. Handing that in cron would likely be on the user end, not the application end.
How is it off topic? All I was suggesting was the ability to automatically download the data export file periodically + an api token to connect without needing to use username/password.
Both this issue #75 and #1012 are both related however they are asking two different things. 1012 is requesting direct migration to another instance (which could also be done through data export), and 75 is requesting a CSV data export to be stored locally.

@TheGameFreak720 literally said
>work on this since I heavily need a CSV export to have backups stored just in case.

I was simply adding the suggestion of additionally providing users a basic script they could use, and api token for users to automatically grab these exports instead of needing to do it manually (and risk having outdated exports). The script could then be placed on any system (like cron) of the users choice. 

I fail to see how my suggestion is off topic at all.

Thank you for the clarification - I thought you were referring to backing up user account data that could be used to move instances specifically (so, more like the data you would need to migrate accounts, as opposed to the type of data you'd get from a Goodreads csv export, which has the books you've catalogued, but not other information about your account).

Adding support for API keys would be a first step to helping users automate tasks like backing up their data via CSV export, and you are right that that hasn't been implemented yet.

Apologies for the confusion - I didn't mean to be harsh, I'm just trying to keep individual issues pretty granular and focused so that the scope of the work is clearer.
To talk for myself, I meant just a button that I can get a CSV export because that is the bare minimum I need. Would be nice for automatic backups, but that gets complicated quickly and in my view out of scope for this task. 
That being said, in an alternative issue I have worked with queue servers using Redis and can give suggestions on how to do it. For myself, I rather work on this and some UI as long as @mouse-reeve approves.
For sure @TheGameFreak720 - please work on what you're interested in
I've been trying to get this app to run between last night and today. I fixed most of the docker and docker-compose setup. The way I did it was using these 2 guides:

* https://docs.docker.com/engine/install/fedora/
* https://docs.docker.com/engine/install/linux-postinstall/

Then docker-compose: https://developer.fedoraproject.org/tools/docker/compose.html

Only thing I did there was install the package. 

I was able to run [ArchiveBox](https://github.com/ArchiveBox/ArchiveBox) that I believe is a Django app as well. I'm still running into issues with this one, but different ones:

```
[zero@skeletor bookwyrm]$ docker-compose run --rm web python manage.py migrate
Starting bookwyrm_redis_broker_1 ... 
Starting bookwyrm_redis_activity_1 ... error
Starting bookwyrm_redis_broker_1   ... error

ERROR: for bookwyrm_redis_activity_1  Cannot start service redis_activity: OCI runtime create failed: selinux label is specified in config, but selinux is disabled or not supported: unknown

Starting bookwyrm_db_1             ... error

ERROR: for bookwyrm_db_1  Cannot start service db: OCI runtime create failed: selinux label is specified in config, but selinux is disabled or not supported: unknown

ERROR: for redis_activity  Cannot start service redis_activity: OCI runtime create failed: selinux label is specified in config, but selinux is disabled or not supported: unknown

ERROR: for redis_broker  Cannot start service redis_broker: OCI runtime create failed: selinux label is specified in config, but selinux is disabled or not supported: unknown

ERROR: for db  Cannot start service db: OCI runtime create failed: selinux label is specified in config, but selinux is disabled or not supported: unknown
ERROR: Encountered errors while bringing up the project.

```

If you know what this could be I would appreciate it. I haven't found much on this specifically
This forum post says it's not possible, but I have been digging and SE-Linux currently is disabled hence why these errors are popping up. I would need to enable this service in order to hopefully get the app running. Will try tomorrow morning

* https://serverfault.com/questions/757606/how-to-enable-selinux-inside-of-a-centos-docker-container
I haven't forgotten about this. Just been busy with work and personal. 
Unfortunately, if I'm using Fedora it is not possible to run containers that have an SELinux policy. Seems to me that is more of [an enterprise feature for CentOS](https://www.arhea.net/posts/2020-04-28-selinux-for-containers.html) for having policies in containers. You can [enable SELinux](https://docs.fedoraproject.org/en-US/quick-docs/getting-started-with-selinux/), but I don't know too much about it to make that judgement in my local machine. This seems to be a feature for VM's exclusively, and one of the blog posts above says that. I'm very unsure this will be possible with Fedora and might have to allow someone else to take it until I get another Distro since virtual machines can be a pain to code in and more in my not so powerful laptop.
podman maybe?
Maybe, I've never used it. I can look into it just that I'm a beginner in Docker and this feels like a deep end where I could make my machine vulnerable if not done correctly.
Kind of embarrassing how easy this was on the thing we are replacing (goodreads) but isn't yet possible here
@WorldsEndless I hear how important being able to download user data is as a way to give users ownership and agency over their data; it sounds like this feature is something you value, and that you're finding it frustrating that it hasn't been added yet.

I'm glad that you're sharing that this is important to you, and it's helpful to hear that it's a pain point, but I need to you express that in a way that is considerate of the people contributing to the project. When you phrase your concern in a way that, to me, reads as snarky or sarcastic, it feels pretty demoralizing. This is a large project with a very small number of contributors, all of whom are working in their free time out of enthusiasm for the project, and as a result, the amount of work that needs to be done can feel overwhelming. Having our work called embarrassing is hurtful, and it isn't a constructive way to provide feedback.

I think that we're all coming to this from a shared desire to improve and expand BookWyrm, and I hope we can also bring a shared understanding that everyone involved is doing their best under challenging circumstances with the time and attention available to them.
My sincere apologies. I am not familiar with the culture or workforce of this projert and sometimes forget that I am not talking to one of the big for-profit corprations that might be ignoring this point deliberately. I'm glad to hear that any quietude of this issue is due to resource limitations, not to lack of interest or it not fitting into some business-plan.

Although Python is not my strongest language, I'm happy to help however I may. Thanks for your reasonable and well-spoken remonstration.

- Tory

Thank you so much! I'm super happy to be useful however I can with code contributions and there's absolutely no expectation that anyone is a pro at it; if there's a task you'd be interest in taking on, let me know how I can support :)

Resolving patch:
diff --git a/bookwyrm/urls.py b/bookwyrm/urls.py
--- a/bookwyrm/urls.py
+++ b/bookwyrm/urls.py
@@ -475,6 +475,12 @@
         views.ChangePassword.as_view(),
         name="prefs-password",
     ),
+    re_path(r"^preferences/export/?$", views.Export.as_view(), name="prefs-export"),
+    re_path(
+        r"^preferences/export/file/?$",
+        views.export_user_book_data,
+        name="prefs-export-file",
+    ),
     re_path(r"^preferences/delete/?$", views.DeleteUser.as_view(), name="prefs-delete"),
     re_path(r"^preferences/block/?$", views.Block.as_view(), name="prefs-block"),
     re_path(r"^block/(?P<user_id>\d+)/?$", views.Block.as_view()),
diff --git a/bookwyrm/views/__init__.py b/bookwyrm/views/__init__.py
--- a/bookwyrm/views/__init__.py
+++ b/bookwyrm/views/__init__.py
@@ -28,6 +28,7 @@
 # user preferences
 from .preferences.change_password import ChangePassword
 from .preferences.edit_user import EditUser
+from .preferences.export import Export, export_user_book_data
 from .preferences.delete_user import DeleteUser
 from .preferences.block import Block, unblock
 
diff --git a/bookwyrm/views/preferences/export.py b/bookwyrm/views/preferences/export.py
new file mode 100644
--- /dev/null
+++ b/bookwyrm/views/preferences/export.py
@@ -0,0 +1,97 @@
+""" Let users export their book data """
+import csv
+
+from django.contrib.auth.decorators import login_required
+from django.db.models import Q
+from django.http import StreamingHttpResponse
+from django.template.response import TemplateResponse
+from django.views import View
+from django.utils.decorators import method_decorator
+from django.views.decorators.http import require_GET
+
+from bookwyrm import models
+
+# pylint: disable=no-self-use
+@method_decorator(login_required, name="dispatch")
+class Export(View):
+    """Let users export data"""
+
+    def get(self, request):
+        """Request csv file"""
+        return TemplateResponse(request, "preferences/export.html")
+
+
+@login_required
+@require_GET
+def export_user_book_data(request):
+    """Streaming the csv file of a user's book data"""
+    data = (
+        models.Edition.viewer_aware_objects(request.user)
+        .filter(
+            Q(shelves__user=request.user)
+            | Q(readthrough__user=request.user)
+            | Q(review__user=request.user)
+            | Q(comment__user=request.user)
+            | Q(quotation__user=request.user)
+        )
+        .distinct()
+    )
+
+    generator = csv_row_generator(data, request.user)
+
+    pseudo_buffer = Echo()
+    writer = csv.writer(pseudo_buffer)
+    # for testing, if you want to see the results in the browser:
+    # from django.http import JsonResponse
+    # return JsonResponse(list(generator), safe=False)
+    return StreamingHttpResponse(
+        (writer.writerow(row) for row in generator),
+        content_type="text/csv",
+        headers={"Content-Disposition": 'attachment; filename="bookwyrm-export.csv"'},
+    )
+
+
+def csv_row_generator(books, user):
+    """generate a csv entry for the user's book"""
+    deduplication_fields = [
+        f.name
+        for f in models.Edition._meta.get_fields()  # pylint: disable=protected-access
+        if getattr(f, "deduplication_field", False)
+    ]
+    fields = (
+        ["title", "author_text"]
+        + deduplication_fields
+        + ["rating", "review_name", "review_cw", "review_content"]
+    )
+    yield fields
+    for book in books:
+        # I think this is more efficient than doing a subquery in the view? but idk
+        review_rating = (
+            models.Review.objects.filter(user=user, book=book, rating__isnull=False)
+            .order_by("-published_date")
+            .first()
+        )
+
+        book.rating = review_rating.rating if review_rating else None
+
+        review = (
+            models.Review.objects.filter(user=user, book=book, content__isnull=False)
+            .order_by("-published_date")
+            .first()
+        )
+        if review:
+            book.review_name = review.name
+            book.review_cw = review.content_warning
+            book.review_content = review.raw_content
+        yield [getattr(book, field, "") or "" for field in fields]
+
+
+class Echo:
+    """An object that implements just the write method of the file-like
+    interface. (https://docs.djangoproject.com/en/3.2/howto/outputting-csv/)
+    """
+
+    # pylint: disable=no-self-use
+    def write(self, value):
+        """Write the value by returning it, instead of storing in a buffer."""
+        return value


Test patch:
diff --git a/bookwyrm/tests/views/test_export.py b/bookwyrm/tests/views/test_export.py
new file mode 100644
--- /dev/null
+++ b/bookwyrm/tests/views/test_export.py
@@ -0,0 +1,69 @@
+""" test for app action functionality """
+from unittest.mock import patch
+
+from django.http import StreamingHttpResponse
+from django.test import TestCase
+from django.test.client import RequestFactory
+
+from bookwyrm import models, views
+from bookwyrm.tests.validate_html import validate_html
+
+
+@patch("bookwyrm.activitystreams.add_book_statuses_task.delay")
+@patch("bookwyrm.activitystreams.add_status_task.delay")
+@patch("bookwyrm.activitystreams.populate_stream_task.delay")
+@patch("bookwyrm.suggested_users.rerank_suggestions_task.delay")
+@patch("bookwyrm.models.activitypub_mixin.broadcast_task.apply_async")
+class ExportViews(TestCase):
+    """viewing and creating statuses"""
+
+    def setUp(self):
+        """we need basic test data and mocks"""
+        self.factory = RequestFactory()
+        with patch("bookwyrm.suggested_users.rerank_suggestions_task.delay"), patch(
+            "bookwyrm.activitystreams.populate_stream_task.delay"
+        ):
+            self.local_user = models.User.objects.create_user(
+                "mouse@local.com",
+                "mouse@mouse.com",
+                "mouseword",
+                local=True,
+                localname="mouse",
+                remote_id="https://example.com/users/mouse",
+            )
+        self.work = models.Work.objects.create(title="Test Work")
+        self.book = models.Edition.objects.create(
+            title="Test Book",
+            remote_id="https://example.com/book/1",
+            parent_work=self.work,
+            isbn_13="9781234567890",
+            bnf_id="beep",
+        )
+
+    def tst_export_get(self, *_):
+        """request export"""
+        request = self.factory.get("")
+        request.user = self.local_user
+        result = views.Export.as_view()(request)
+        validate_html(result.render())
+
+    def test_export_file(self, *_):
+        """simple export"""
+        models.ShelfBook.objects.create(
+            shelf=self.local_user.shelf_set.first(),
+            user=self.local_user,
+            book=self.book,
+        )
+        request = self.factory.get("")
+        request.user = self.local_user
+        export = views.export_user_book_data(request)
+        self.assertIsInstance(export, StreamingHttpResponse)
+        self.assertEqual(export.status_code, 200)
+        result = list(export.streaming_content)
+        # pylint: disable=line-too-long
+        self.assertEqual(
+            result[0],
+            b"title,author_text,remote_id,openlibrary_key,inventaire_id,librarything_key,goodreads_key,bnf_id,viaf,wikidata,asin,isbn_10,isbn_13,oclc_number,rating,review_name,review_cw,review_content\r\n",
+        )
+        expected = f"Test Book,,{self.book.remote_id},,,,,beep,,,,123456789X,9781234567890,,,,,\r\n"
+        self.assertEqual(result[1].decode("utf-8"), expected)