internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: google/turbinia

Issue:
[Bug]: plaso jobs fail when plaso >6 months old
### What steps will reproduce the bug?

* Use a plaso version > 6 months old
* Run a plaso job
* pinfo.py will output warnings that are not json format so evidence.py will error
* request never completes

### What is the expected behavior?

requests to end...
pinfo.py output to only contain json and/or we verify

### What do you see instead?

```
output of pinfo.py

>>> print(t.splitlines())
['WARNING: the version of plaso you are using is more than 6 months old. We', 'strongly recommend to update it.', '', '{"storage_counters": {"parsers": {"filestat": 897, "total": 897}, "event_labels": {}}}']
So the relevant line is always the last line which is the json we need.
```


### Additional information

_No response_


Discussion:

Resolving patch:
diff --git a/turbinia/evidence.py b/turbinia/evidence.py
--- a/turbinia/evidence.py
+++ b/turbinia/evidence.py
@@ -1070,6 +1070,8 @@ def _validate(self):
       log.info(f'Running pinfo.py to validate PlasoFile {self.local_path}')
       command = subprocess.run(cmd, capture_output=True, check=True)
       storage_counters_json = command.stdout.decode('utf-8').strip()
+      # pinfo.py might print warnings (non-json) so we only need to last line of output
+      storage_counters_json = storage_counters_json.splitlines()[-1]
       storage_counters = json.loads(storage_counters_json)
       total_file_events = storage_counters.get('storage_counters', {}).get(
           'parsers', {}).get('total', 0)


Test patch:
Repository: bookwyrm-social/bookwyrm

Issue:
Deployment flow
This probably means dockerization??


Discussion:
I began writing a dockerfile for fedireads last weekend because I was having some issues with postgres on ubuntu, I could finish it up this weekend for you to have a look at if you want @mouse-reeve ?
it would be greatly appreciated!

Resolving patch:
diff --git a/fedireads/__init__.py b/fedireads/__init__.py
--- a/fedireads/__init__.py
+++ b/fedireads/__init__.py
@@ -0,0 +1,8 @@
+from __future__ import absolute_import, unicode_literals
+
+# This will make sure the app is always imported when
+# Django starts so that shared_task will use this app.
+from .celery import app as celery_app
+
+__all__ = ('celery_app',)
+
diff --git a/fedireads/celery.py b/fedireads/celery.py
new file mode 100644
--- /dev/null
+++ b/fedireads/celery.py
@@ -0,0 +1,24 @@
+from __future__ import absolute_import, unicode_literals
+
+import os
+
+from celery import Celery
+
+# set the default Django settings module for the 'celery' program.
+os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'fedireads.settings')
+
+app = Celery('fedireads')
+
+# Using a string here means the worker doesn't have to serialize
+# the configuration object to child processes.
+# - namespace='CELERY' means all celery-related configuration keys
+#   should have a `CELERY_` prefix.
+app.config_from_object('django.conf:settings', namespace='CELERY')
+
+# Load task modules from all registered Django app configs.
+app.autodiscover_tasks()
+
+
+@app.task(bind=True)
+def debug_task(self):
+    print('Request: {0!r}'.format(self.request))
diff --git a/fedireads/settings.py b/fedireads/settings.py
--- a/fedireads/settings.py
+++ b/fedireads/settings.py
@@ -17,11 +17,16 @@
 # SECURITY WARNING: don't run with debug turned on in production!
 DEBUG = env.bool('DEBUG', True)
 
-# TODO: annoying that I keep changing and re-commiting this
 DOMAIN = env('DOMAIN')
 ALLOWED_HOSTS = env.list('ALLOWED_HOSTS', ['*'])
 OL_URL = env('OL_URL')
 
+# celery/rebbitmq
+CELERY_BROKER_URL = env('CELERY_BROKER')
+CELERY_ACCEPT_CONTENT = ['json']
+CELERY_TASK_SERIALIZER = 'json'
+CELERY_RESULT_BACKEND = 'amqp'
+
 # Application definition
 
 INSTALLED_APPS = [
@@ -33,6 +38,7 @@
     'django.contrib.staticfiles',
     'django.contrib.humanize',
     'fedireads',
+    'celery',
 ]
 
 MIDDLEWARE = [
@@ -75,10 +81,10 @@
 FEDIREADS_DBS = {
     'postgres': {
         'ENGINE': 'django.db.backends.postgresql_psycopg2',
-        'NAME': 'fedireads',
-        'USER': 'fedireads',
-        'PASSWORD': 'fedireads',
-        'HOST': '',
+        'NAME': env('POSTGRES_DB', 'fedireads'),
+        'USER': env('POSTGRES_USER', 'fedireads'),
+        'PASSWORD': env('POSTGRES_PASSWORD', 'fedireads'),
+        'HOST': 'db',
         'PORT': 5432
     },
     'sqlite': {
@@ -91,6 +97,7 @@
     'default': FEDIREADS_DBS[FEDIREADS_DATABASE_BACKEND]
 }
 
+
 LOGIN_URL = '/login/'
 AUTH_USER_MODEL = 'fedireads.User'
 


Test patch:
Repository: bookwyrm-social/bookwyrm

Issue:
Error handling for goodreads import files
It 500s if you try to upload, say, a jpg


Discussion:
Absolutely true.

I'd like to look at making import asynchronous via celery first because I think that will change the workflow quite a bit here. And I'm not sure I want this to be the first thing to use celery because we should start with something simpler.

Resolving patch:
diff --git a/fedireads/goodreads_import.py b/fedireads/goodreads_import.py
--- a/fedireads/goodreads_import.py
+++ b/fedireads/goodreads_import.py
@@ -14,6 +14,8 @@
 def create_job(user, csv_file):
     job = ImportJob.objects.create(user=user)
     for index, entry in enumerate(list(csv.DictReader(csv_file))[:MAX_ENTRIES]):
+        if not all(x in entry for x in ('ISBN13', 'Title', 'Author')):
+            raise ValueError("Author, title, and isbn must be in data.")
         ImportItem(job=job, index=index, data=entry).save()
     return job
 
diff --git a/fedireads/view_actions.py b/fedireads/view_actions.py
--- a/fedireads/view_actions.py
+++ b/fedireads/view_actions.py
@@ -418,10 +418,15 @@ def import_data(request):
     ''' ingest a goodreads csv '''
     form = forms.ImportForm(request.POST, request.FILES)
     if form.is_valid():
-        job = goodreads_import.create_job(
-            request.user,
-            TextIOWrapper(request.FILES['csv_file'], encoding=request.encoding)
-        )
+        try:
+            job = goodreads_import.create_job(
+                request.user,
+                TextIOWrapper(
+                    request.FILES['csv_file'],
+                    encoding=request.encoding)
+            )
+        except (UnicodeDecodeError, ValueError):
+            return HttpResponseBadRequest('Not a valid csv file')
         goodreads_import.start_import(job)
         return redirect('/import_status/%d' % (job.id,))
     return HttpResponseBadRequest()


Test patch:
Repository: bookwyrm-social/bookwyrm

Issue:
nginx configuration



Discussion:

Resolving patch:
diff --git a/fedireads/settings.py b/fedireads/settings.py
--- a/fedireads/settings.py
+++ b/fedireads/settings.py
@@ -138,6 +138,8 @@
 # Static files (CSS, JavaScript, Images)
 # https://docs.djangoproject.com/en/2.0/howto/static-files/
 
+PROJECT_DIR = os.path.dirname(os.path.abspath(__file__))
 STATIC_URL = '/static/'
+STATIC_ROOT = os.path.join(BASE_DIR, env('STATIC_ROOT', 'static'))
 MEDIA_URL = '/images/'
-MEDIA_ROOT = os.path.join(BASE_DIR, env('MEDIA_ROOT', 'images/'))
+MEDIA_ROOT = os.path.join(BASE_DIR, env('MEDIA_ROOT', 'images'))


Test patch:
Repository: bookwyrm-social/bookwyrm

Issue:
change password



Discussion:

Resolving patch:
diff --git a/bookwyrm/forms.py b/bookwyrm/forms.py
--- a/bookwyrm/forms.py
+++ b/bookwyrm/forms.py
@@ -25,6 +25,8 @@ def __init__(self, *args, **kwargs):
                 input_type = visible.field.widget.input_type
             if isinstance(visible.field.widget, Textarea):
                 input_type = 'textarea'
+                visible.field.widget.attrs['cols'] = None
+                visible.field.widget.attrs['rows'] = None
             visible.field.widget.attrs['class'] = css_classes[input_type]
 
 class LoginForm(CustomForm):
@@ -96,7 +98,9 @@ class Meta:
 class EditUserForm(CustomForm):
     class Meta:
         model = models.User
-        fields = ['avatar', 'name', 'summary', 'manually_approves_followers']
+        fields = [
+            'avatar', 'name', 'email', 'summary', 'manually_approves_followers'
+        ]
         help_texts = {f: None for f in fields}
 
 
diff --git a/bookwyrm/urls.py b/bookwyrm/urls.py
--- a/bookwyrm/urls.py
+++ b/bookwyrm/urls.py
@@ -85,6 +85,7 @@
     re_path(r'^user-register/?$', actions.register),
     re_path(r'^reset-password-request/?$', actions.password_reset_request),
     re_path(r'^reset-password/?$', actions.password_reset),
+    re_path(r'^change-password/?$', actions.password_change),
 
     re_path(r'^edit_profile/?$', actions.edit_profile),
 
diff --git a/bookwyrm/view_actions.py b/bookwyrm/view_actions.py
--- a/bookwyrm/view_actions.py
+++ b/bookwyrm/view_actions.py
@@ -106,9 +106,8 @@ def password_reset_request(request):
     return TemplateResponse(request, 'password_reset_request.html', data)
 
 
-
 def password_reset(request):
-    ''' allow a user to change their password '''
+    ''' allow a user to change their password through an emailed token '''
     try:
         reset_code = models.PasswordReset.objects.get(
             code=request.POST.get('reset-code')
@@ -133,6 +132,21 @@ def password_reset(request):
     return redirect('/')
 
 
+@login_required
+def password_change(request):
+    ''' allow a user to change their password '''
+    new_password = request.POST.get('password')
+    confirm_password = request.POST.get('confirm-password')
+
+    if new_password != confirm_password:
+        return redirect('/user-edit')
+
+    request.user.set_password(new_password)
+    request.user.save()
+    login(request, request.user)
+    return redirect('/user-edit')
+
+
 @login_required
 def edit_profile(request):
     ''' les get fancy with images '''


Test patch:
Repository: bookwyrm-social/bookwyrm

Issue:
Instance settings
there should be a json file with local settings with (non-exhaustively):

- instance name
- domain
- instance description
- code of conduct
- allow registration vs invite only
- suggested book tags



Discussion:
I think the remaining work before this is close-able is to create the admin panel to set these. Uploading logos and themes would be the next level, for some other issue.

Resolving patch:
diff --git a/bookwyrm/admin.py b/bookwyrm/admin.py
new file mode 100644
--- /dev/null
+++ b/bookwyrm/admin.py
@@ -0,0 +1,6 @@
+''' models that will show up in django admin for superuser '''
+from django.contrib import admin
+from bookwyrm import models
+
+admin.site.register(models.SiteSettings)
+admin.site.register(models.User)


Test patch:
Repository: bookwyrm-social/bookwyrm

Issue:
null value in column "deleted_date" of relation "bookwyrm_status" violates non-null constraint error
**I see:**
A few variations of the following fullscreen Django error:
```
IntegrityError at /<the-route-hit>/
null value in column "deleted_date" of relation "bookwyrm_status" violates non-null constraint error
```

**When I:**
- toggle the state of the "start reading" toggle
- click post on a comment
- click post on a quote
- click post on a review

**My Current Theories / Questions**
My guess is that the DB is expecting `not null` on `is_deleted` but the state of the book doesn't default to `is_deleted` as `false`. I need to dig more into the data model to understand which part of the schema is having this issue. It could also be that a migration didn't fully run? Wanted to flag this before I dig into it in case it's a small thing that I'm missing since I'm new to the code

**Context**
This happens on a local env installed from the Docker instructions.


Discussion:
Oh I bet I know what happened. I initially wrote the deleted date to have a default value of "now" and then the next day realized that makes no sense, but I forgot to change the model field to be null-able and blank-able. This should be a straightforward change in the Status model

Resolving patch:
diff --git a/bookwyrm/migrations/0056_auto_20201021_0150.py b/bookwyrm/migrations/0056_auto_20201021_0150.py
new file mode 100644
--- /dev/null
+++ b/bookwyrm/migrations/0056_auto_20201021_0150.py
@@ -0,0 +1,18 @@
+# Generated by Django 3.0.7 on 2020-10-21 01:50
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('bookwyrm', '0055_merge_20201017_0011'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='status',
+            name='deleted_date',
+            field=models.DateTimeField(blank=True, null=True),
+        ),
+    ]
diff --git a/bookwyrm/models/status.py b/bookwyrm/models/status.py
--- a/bookwyrm/models/status.py
+++ b/bookwyrm/models/status.py
@@ -23,7 +23,7 @@ class Status(OrderedCollectionPageMixin, BookWyrmModel):
     # the created date can't be this, because of receiving federated posts
     published_date = models.DateTimeField(default=timezone.now)
     deleted = models.BooleanField(default=False)
-    deleted_date = models.DateTimeField()
+    deleted_date = models.DateTimeField(blank=True, null=True)
     favorites = models.ManyToManyField(
         'User',
         symmetrical=False,


Test patch:
Repository: bookwyrm-social/bookwyrm

Issue:
post privacy settings



Discussion:
- just me
- direct (mentions only)
- followers only
- unlisted
- public

Resolving patch:
diff --git a/bookwyrm/views.py b/bookwyrm/views.py
--- a/bookwyrm/views.py
+++ b/bookwyrm/views.py
@@ -130,17 +130,22 @@ def get_activity_feed(user, filter_level, model=models.Status):
     if filter_level in ['friends', 'home']:
         # people you follow and direct mentions
         activities = activities.filter(
-            Q(user__in=following, privacy='public') | \
-                Q(mention_users=user)
+            Q(user__in=following, privacy__in=['public', 'unlisted', 'followers']) | \
+                Q(mention_users=user) | Q(user=user)
         )
     elif filter_level == 'self':
         activities = activities.filter(user=user, privacy='public')
     elif filter_level == 'local':
-        # everyone on this instance
-        activities = activities.filter(user__local=True, privacy='public')
+        # everyone on this instance except unlisted
+        activities = activities.filter(
+            Q(user__in=following, privacy='followers') | Q(privacy='public'),
+            user__local=True
+        )
     else:
         # all activities from everyone you federate with
-        activities = activities.filter(privacy='public')
+        activities = activities.filter(
+            Q(user__in=following, privacy='followers') | Q(privacy='public')
+        )
 
     return activities
 
@@ -386,9 +391,14 @@ def status_page(request, username, status_id):
     except ValueError:
         return HttpResponseNotFound()
 
+    # the url should have the poster's username in it
     if user != status.user:
         return HttpResponseNotFound()
 
+    # make sure the user is authorized to see the status
+    if not status_visible_to_user(request.user, status):
+        return HttpResponseNotFound()
+
     if is_api_request(request):
         return JsonResponse(status.to_activity(), encoder=ActivityEncoder)
 
@@ -397,6 +407,19 @@ def status_page(request, username, status_id):
     }
     return TemplateResponse(request, 'status.html', data)
 
+def status_visible_to_user(viewer, status):
+    ''' is a user authorized to view a status? '''
+    if viewer == status.user or status.privacy in ['public', 'unlisted']:
+        return True
+    if status.privacy == 'followers' and \
+            status.user.followers.filter(id=viewer.id).first():
+        return True
+    if status.privacy == 'direct' and \
+            status.mention_users.filter(id=viewer.id).first():
+        return True
+    return False
+
+
 
 @csrf_exempt
 def replies_page(request, username, status_id):


Test patch:
Repository: bookwyrm-social/bookwyrm

Issue:
It's possible to create a FollowRequest for a Follow that already exists
Send a follow request.
User accepts.
Send a second follow request (eg. using an old tab)

Because requests and follows are stored in separate tables it's not possible to enforce this with a constraint. Pre-checking would leave a race condition, so I guess we either need to mess with table locks or create the follow request, and then delete it if its unneeded. Can we enforce this in the pre save of the model, perhaps?


Discussion:

Resolving patch:
diff --git a/bookwyrm/models/relationship.py b/bookwyrm/models/relationship.py
--- a/bookwyrm/models/relationship.py
+++ b/bookwyrm/models/relationship.py
@@ -82,6 +82,17 @@ class UserFollowRequest(UserRelationship):
     ''' following a user requires manual or automatic confirmation '''
     status = 'follow_request'
 
+    def save(self, *args, **kwargs):
+        ''' make sure the follow relationship doesn't already exist '''
+        try:
+            UserFollows.objects.get(
+                user_subject=self.user_subject,
+                user_object=self.user_object
+            )
+            return None
+        except UserFollows.DoesNotExist:
+            return super().save(*args, **kwargs)
+
 
 class UserBlocks(UserRelationship):
     ''' prevent another user from following you and seeing your posts '''


Test patch:
Repository: bookwyrm-social/bookwyrm

Issue:
Remote follow silently failing
When I try to follow a remote user, the remote user is not notified and the relationship is not confirmed. No errors show up locally that I've found.


Discussion:
this is fully working between bookwyrm instances now, still seeing glitchiness following mastodon users

Resolving patch:
diff --git a/bookwyrm/broadcast.py b/bookwyrm/broadcast.py
--- a/bookwyrm/broadcast.py
+++ b/bookwyrm/broadcast.py
@@ -61,7 +61,7 @@ def broadcast_task(sender_id, activity, recipients):
     return errors
 
 
-def sign_and_send(sender, activity, destination):
+def sign_and_send(sender, data, destination):
     ''' crpyto whatever and http junk '''
     now = http_date()
 
@@ -69,7 +69,6 @@ def sign_and_send(sender, activity, destination):
         # this shouldn't happen. it would be bad if it happened.
         raise ValueError('No private key found for sender')
 
-    data = json.dumps(activity).encode('utf-8')
     digest = make_digest(data)
 
     response = requests.post(
diff --git a/bookwyrm/signatures.py b/bookwyrm/signatures.py
--- a/bookwyrm/signatures.py
+++ b/bookwyrm/signatures.py
@@ -44,7 +44,8 @@ def make_signature(sender, destination, date, digest):
 
 def make_digest(data):
     ''' creates a message digest for signing '''
-    return 'SHA-256=' + b64encode(hashlib.sha256(data).digest()).decode('utf-8')
+    return 'SHA-256=' + b64encode(hashlib.sha256(data.encode('utf-8'))\
+                        .digest()).decode('utf-8')
 
 
 def verify_digest(request):


Test patch:
Repository: bookwyrm-social/bookwyrm

Issue:
Issues with favoriting / unfavoriting status
I am logged in on bookwyrm.social. I tried liking a post about how salt marsh wants to read Blockchain Chicken Farm on:

https://bookwyrm.social/user/mouse

When I clicked "like", it didn't immediately change the heart green. When I refreshed the page, it was green. Clicking the heart again sends a request to https://bookwyrm.social/unfavorite/448 which 200s, however when I look at the response it's an HTML page that says among other things

```
<div class="block">
    <h2 class="title">Server Error</h2>
    <p>Something went wrong! Sorry about that.</p>
</div>
```

Similarly, if I browse to a note on that page: https://bookwyrm.social/user/mouse/generatednote/448

If I click the heart on the original post, it sends a POST request that doesn't seem to have any form data to https://bookwyrm.social/unfavorite/ which 200s but the body is HTML that says a server error happened.


Discussion:

Resolving patch:
diff --git a/bookwyrm/incoming.py b/bookwyrm/incoming.py
--- a/bookwyrm/incoming.py
+++ b/bookwyrm/incoming.py
@@ -269,7 +269,12 @@ def handle_favorite(activity):
 @app.task
 def handle_unfavorite(activity):
     ''' approval of your good good post '''
-    like = activitypub.Like(**activity['object']).to_model(models.Favorite)
+    try:
+        like = models.Favorite.objects.filter(
+            remote_id=activity['object']['id']
+        ).first()
+    except models.Favorite.DoesNotExist:
+        return
     like.delete()
 
 
diff --git a/bookwyrm/outgoing.py b/bookwyrm/outgoing.py
--- a/bookwyrm/outgoing.py
+++ b/bookwyrm/outgoing.py
@@ -299,7 +299,8 @@ def handle_unfavorite(user, status):
         # can't find that status, idk
         return
 
-    fav_activity = activitypub.Undo(actor=user, object=favorite)
+    fav_activity = favorite.to_undo_activity(user)
+    favorite.delete()
     broadcast(user, fav_activity, direct_recipients=[status.user])
 
 


Test patch:
Repository: bookwyrm-social/bookwyrm

Issue:
Edit book form fails silently
Often when I try to edit a book, after pressing save, I am sent back to the edit book form without my changes.

The form should indicate errors and preserve changes when reloading the form after a failed submit


Discussion:

Resolving patch:
diff --git a/bookwyrm/migrations/0013_book_origin_id.py b/bookwyrm/migrations/0013_book_origin_id.py
new file mode 100644
--- /dev/null
+++ b/bookwyrm/migrations/0013_book_origin_id.py
@@ -0,0 +1,18 @@
+# Generated by Django 3.0.7 on 2020-11-24 21:42
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('bookwyrm', '0012_attachment'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='book',
+            name='origin_id',
+            field=models.CharField(blank=True, max_length=255, null=True),
+        ),
+    ]
diff --git a/bookwyrm/models/book.py b/bookwyrm/models/book.py
--- a/bookwyrm/models/book.py
+++ b/bookwyrm/models/book.py
@@ -16,7 +16,7 @@
 
 class Book(ActivitypubMixin, BookWyrmModel):
     ''' a generic book, which can mean either an edition or a work '''
-    origin_id = models.CharField(max_length=255, null=True)
+    origin_id = models.CharField(max_length=255, null=True, blank=True)
     # these identifiers apply to both works and editions
     openlibrary_key = models.CharField(max_length=255, blank=True, null=True)
     librarything_key = models.CharField(max_length=255, blank=True, null=True)
diff --git a/bookwyrm/view_actions.py b/bookwyrm/view_actions.py
--- a/bookwyrm/view_actions.py
+++ b/bookwyrm/view_actions.py
@@ -235,7 +235,12 @@ def edit_book(request, book_id):
 
     form = forms.EditionForm(request.POST, request.FILES, instance=book)
     if not form.is_valid():
-        return redirect(request.headers.get('Referer', '/'))
+        data = {
+            'title': 'Edit Book',
+            'book': book,
+            'form': form
+        }
+        return TemplateResponse(request, 'edit_book.html', data)
     form.save()
 
     outgoing.handle_update_book(request.user, book)


Test patch: