Repository: bookwyrm-social/bookwyrm

Issue:
Unable to connect via ActivityPub to Hubzilla and Zap (and other related projects)


Part 1. Follow a Zap user from bookwyrm. I'll use otest@p3.macgirvin.com which is a throw-away test account. 
Entering the address in the search bar reports nothing found. On the Zap side, we can see a webfinger request (which provides the ActivityPub url) but the ActivityPub URL is never fetched. So there may be something Bookwyrm doesn't like or is getting tripped up on in the Webfinger result. This is what we return:

```
{
    "subject": "acct:otest@p3.macgirvin.com",
    "aliases": [
        "https://p3.macgirvin.com/channel/otest",
        "https://p3.macgirvin.com/~otest",
        "https://p3.macgirvin.com/@otest"
    ],
    "properties": {
        "http://webfinger.net/ns/name": "otest",
        "http://xmlns.com/foaf/0.1/name": "otest",
        "https://w3id.org/security/v1#publicKeyPem": "-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAqMl7q1ilafetgIYCtTdq\nDCIUsA3ZFgEwPIUur57Zxg9jBBr+3Pyq2Ez9xJxz7bXptAAffnAFi0t4ree0CnNF\ntPqHLwOA9qphXrvmeuxFMHiEtPcTgivBuH6jUmWvvah0WpG4lecZUkrCNk/Gf6Ni\nTu/BdCLI17omd7IGaEjSuRQNfLA+SgKpVMcdjs8S7RLUmxHfbp4ODGSMHhYFcg00\n7pYHvjgsrSMthjq/SZqXR+FGjm+LRPfEpxZT69CwaUw3C5b2gyLFdXZgYFW8iyE9\n2Cn6MaprcJjiaNQnFf2nbRnyl+0FvMx1VyZ9dO/kw7oBIiDYNQtfOM7kKYcQZSuF\nnXCW7iZHe3fWKU0ICQiN/z7pMByHC7vtG7KShJTdwn7rgIKG8ZhiowxqirhnSFRK\n45WUiG74SFy1GxbL0ZQ1cXxVuO4ZkUpmwmu9CNZ94/ewkcfgFerw8SjORmuW+RwA\ncntU5L4Bup+7PttoDm15mVxzkVylX0Bo8jBkdnQ2vfKOtGe3yGpfwwuPIxWcmTLQ\nZPowJXMbuw9cHgNLP6QMUA19X8+8tOp0JSfirHC6bEN/9DwcG+SmUQ879EuLJcrl\nqr8GYBN146hWsHVbvhOlckTxb3XrC5t1ufFh0q5cBPFufPSFrklagx3CBOTEsaXa\nmzuJvI8+y7jT7N4T4JrIXOMCAwEAAQ==\n-----END PUBLIC KEY-----\n",
        "http://purl.org/zot/federation": "zot6,activitypub"
    },
    "links": [
        {
            "rel": "http://webfinger.net/rel/avatar",
            "type": "image/png",
            "href": "https://p3.macgirvin.com/photo/profile/l/2"
        },
        {
            "rel": "http://webfinger.net/rel/blog",
            "href": "https://p3.macgirvin.com/channel/otest"
        },
        {
            "rel": "http://openid.net/specs/connect/1.0/issuer",
            "href": "https://p3.macgirvin.com"
        },
        {
            "rel": "http://purl.org/zot/protocol/6.0",
            "type": "application/x-zot+json",
            "href": "https://p3.macgirvin.com/channel/otest"
        },
        {
            "rel": "http://purl.org/openwebauth/v1",
            "type": "application/x-zot+json",
            "href": "https://p3.macgirvin.com/owa"
        },
        {
            "rel": "self",
            "type": "application/ld+json; profile=\"https://www.w3.org/ns/activitystreams\"",
            "href": "https://p3.macgirvin.com/channel/otest"
        },
        {
            "rel": "self",
            "type": "application/activity+json",
            "href": "https://p3.macgirvin.com/channel/otest"
        },
        {
            "rel": "http://ostatus.org/schema/1.0/subscribe",
            "template": "https://p3.macgirvin.com/follow?url={uri}"
        }
    ]
}
```

Part 2: Follow the bookwyrm account from Zap. The Bookwyrm account is located and internal records generated. Then we send a Follow activity to Bookwyrm. It looks like this:

```
{
  "@context":[
    "https://www.w3.org/ns/activitystreams",
    "https://w3id.org/security/v1",
    {
      "zot":"https://p3.macgirvin.com/apschema#",
      "toot":"http://joinmastodon.org/ns#",
      "ostatus":"http://ostatus.org#",
      "schema":"http://schema.org#",
      "litepub":"http://litepub.social/ns#",
      "sm":"http://smithereen.software/ns#",
      "conversation":"ostatus:conversation",
      "manuallyApprovesFollowers":"as:manuallyApprovesFollowers",
      "oauthRegistrationEndpoint":"litepub:oauthRegistrationEndpoint",
      "sensitive":"as:sensitive",
      "movedTo":"as:movedTo",
      "copiedTo":"as:copiedTo",
      "alsoKnownAs":"as:alsoKnownAs",
      "EmojiReact":"as:EmojiReact",
      "commentPolicy":"zot:commentPolicy",
      "topicalCollection":"zot:topicalCollection",
      "eventRepeat":"zot:eventRepeat",
      "emojiReaction":"zot:emojiReaction",
      "expires":"zot:expires",
      "directMessage":"zot:directMessage",
      "Category":"zot:Category",
      "replyTo":"zot:replyTo",
      "PropertyValue":"schema:PropertyValue",
      "value":"schema:value",
      "discoverable":"toot:discoverable",
      "wall":"sm:wall"
    }
  ],
  "id":"https://p3.macgirvin.com/follow/174",
  "type":"Follow",
  "actor":"https://p3.macgirvin.com/channel/otest",
  "object":"https://bookwyrm.social/user/macgirvin",
  "to":[
    "https://bookwyrm.social/user/macgirvin"
  ],
  "signature":{
    "type":"RsaSignature2017",
    "nonce":"39a2ff6caec6732ce0cc26e7ca0a3fbad1bf7360a79002912fa9e86d7803033a",
    "creator":"https://p3.macgirvin.com/channel/otest",
    "created":"2021-10-22T21:09:48Z",
    "signatureValue":"RJXnKsGEpniDE3VmEVMC4TkC3d1A+BIF7LUKNHFymcqpqXv+A4FfZB2dmpEctc2s8xohrlzcVYbjwrVzSptHjTnOY5Sf4yUJ3VLB/qVDHq2ClvJ1KUYsmszLKBixoVCgB80NQTv7RC0uw190Jn6JmXIoJ0C7g5jxwYxxSFwhyKPjWzjLCpLF+LlXhoO+njmf+RcSZq7LHIgMB7Jwqcv8CKyE3KX/JsJX7Pz7pg+RvYuisWuk6frUlfMZpypJX3fatoHwH/hsZs6b+7p9NSaY7ch9TdT4czlpFku09f/x5RXXaozuVM1AKYB/grcQDFvYq0uW/Gn9SWbfgedSAKbSJhGn5Se2RqWrQXkoNUzX8HvmED9g5F4GRHRcLmXKp2WoGY03AAqXnmxfE13a5DkibvI0D1RtfJ33Yz7AC2eyS2h93Bb1ex9IebsGedXwLV7gxRNUR+Pq4/NKKZIWWH7JDHSKNQy733n5v9W0fcCMsycbgUlhvdCA+phBEdasyblfqnNlkzg+W4rd1aoGBSJ8JkSpeX17SHFs1897Vsv+rSb/289uaOBjVXMNSjelxa/b3PlXnogm5zpwhCyJBL13DhNo/HVCvEAb2r4iMbT2ZzUgPCyz75hiU358XGEL4QklSlcfcRDO36X0Q5psx+tsC5ebaFaSA+r6HnT1xd38ixs="
  }
}
```

This is posted to the bookwyrm account inbox and is rejected by bookwyrm with a 401 response. 

I am aware of the open issue with regard to projects which enforce HTTP signatures and this will ultimately present an issue with communication to our projects, but I'm just trying to get past the initial connection hurdles at the moment. Any help would be appreciated.  The macgirvin account is on bookwyrm.social and these activities *might* still be present in the logs. 



Discussion:
Looking at code now - it appears that fetch requests are sent with Accept header of "application/json" which will likely fail on our systems in spectacular ways because we serve multiple types of JSON depending on the protocol implementation. 

The ActivityPub spec is quite specific about this in section 3.2. Recommend using:

Accept: application/activity+json, application/ld+json; profile="https://www.w3.org/ns/activitystreams"

You are free to tack application/json on the end if that keeps things from breaking on the bookwyrm side, but at least we'll be able to respond correctly to ActivityPub  and ActivityStreams fetches (which use a slightly different convention). I can supply a patch if desired, but as this is my first look at the code I'd probably prefer this be reviewed by somebody with some experience in this code base. 

bookwyrm/connectors/abstract_connector.py::L231

```
- "Accept": "application/json; charset=utf-8",
+ "Accept": "application/activity+json, application/ld+json; profile=\"https://www.w3.org/ns/activitystreams\", application/json; charset=utf-8",
```



This should have been a simple one-line fix. Since I don't know the code style and formatting that is required for this codebase I will not be attempting any further merge requests. 
I'm sorry that was a frustrating process for you! Your work in that pull request looked great and I'm grateful that you went to the effort to fix the issue. If you're down to re-open the pull request, I would be happy to merge it and resolve the style issues myself, or work through them with you if you prefer. There's more info on the formatting and style guides here: https://docs.joinbookwyrm.com/style-guide.html, but it isn't intended to be a gatekeeping system or a hinderance towards opening a PR
I can't tell if it's acceptable now or not. It complained about indentation (which was supplied automatically by emacs) and since I don't know if you're using tabs or spaces I switched it to tabs and put it at indent "4" (tabs this time) as suggested by pylint. This appears to look correct in a terminal but looks bloody awful on github and doesn't match the current indentation at all, so I suspect you're using spaces and this will also fail although 4 spaces likewise doesn't match the current block indent level. So I guess it's all about adding and remove random whitespace and praying it eventually turns green since I've got nothing more to go on. I've run out of time to do anything more today. 

.bw-dev black failed with 

```
./bw-dev: line 12: .env: No such file or directory
Failed to load configuration! You may need to update your .env and quote values with special characters in them.
```

I am unable to install/use pylint on my shared hosting plan. 
You should be able to fix the issue with `bw-dev` by copying `.env.dev.example` to `.env`.

It is using 4 space tabs, but the ins and outs of how much indentation is wants on multi-line strings is weird and complicated! Let's definitely find a different solution than you having to trial and error whitespace, which sounds incredibly frustrating. 

When I ran black, this is the result I got:

``` python3
            headers={
                "Accept": (
                    "application/activity+json,"
                    ' application/ld+json; profile="https://www.w3.org/ns/activitystreams",'
                    " application/json; charset=utf-8"
                ),
                "User-Agent": settings.USER_AGENT,
            },
```

Pylint will still produce a line-too-long error on that, but I think pylint is wrong in this case, so you can disable the warning by adding a comment:

``` python3
            headers={  # pylint: disable=line-too-long
                "Accept": (
                ...
```

That should gets the checks passing.
Really truly not worth the effort. 
I'd like to merge your change in, would you be comfortable with me merging it as-is and fixing the whitespace myself? To do that I would need you to re-open the PR. 

Alternatively, I can make the change myself and list you as a co-author. 

Resolving patch:
diff --git a/bookwyrm/connectors/abstract_connector.py b/bookwyrm/connectors/abstract_connector.py
--- a/bookwyrm/connectors/abstract_connector.py
+++ b/bookwyrm/connectors/abstract_connector.py
@@ -227,8 +227,12 @@ def get_data(url, params=None, timeout=10):
         resp = requests.get(
             url,
             params=params,
-            headers={
-                "Accept": "application/json; charset=utf-8",
+            headers={  # pylint: disable=line-too-long
+                "Accept": (
+                    "application/activity+json,"
+                    ' application/ld+json; profile="https://www.w3.org/ns/activitystreams",'
+                    " application/json; charset=utf-8"
+                ),
                 "User-Agent": settings.USER_AGENT,
             },
             timeout=timeout,


Test patch: