Repository: nanos/FediFetcher

Issue:
Fetch parent posts?
I don't know if activitypub carries information about "parent" posts, but just like replies are interesting to fetch, such parent posts are also interesting in order to reconstruct threads.

If such information is carried it should be "easy".. If it's not in the post metadata, I suppose parent post(s) will need to be found from the origin instances - which will make this much harder to achieve.

And either way this whole thing still poses questions about how to handle recursion and such.

That said - has this been considered? Is it ludicrous to even suggest? :)

Discussion:
@nanos: Actually we don't need to fetch the parent posts: Mastodon should normally (more about the exceptions later) do that already.

However, FediFetcher does already fetch replies to all parents and children recursively as it stands.

There are some exceptions where Mastodon cannot fetch parents, particularly when the parent has been deleted, or has different visibility rules. FediFetcher does not try to circumvent any such exceptions, as this would be seen as unethical.@ltning: But .. but .. There are public posts on my timeline that are clearly responses to posts elsewhere, and those also seem to be public (I can see them when I open the home timeline of the reply I'm seeing).

Example: https://mastodon.gamedev.place/@lritter/114137433881270060 - I only get @lritter's reply (the last one in the thread at the time of this writing); none of the preceding posts.

I'm running Pleroma, not Mastodon - not sure if that has anything to do with it?@nanos: Hm. Interesting. Need to look into that.@ltning: What's strange is that according to the log, at least the direct parent of the post by lritter has been fetched: `2025-03-10 10:18:41 UTC: Added context url https://mastodon.social/@dotstdy/114137189520244929` ..

While probably unrelated, I'm also seeing this:
```
2025-03-09 23:19:07 UTC: Added context url https://friedcheese.us/objects/166bc571-3b47-4806-94c0-c003b86686a8           
2025-03-09 23:19:08 UTC: Added context url https://friedcheese.us/objects/5c3230c7-b25f-4ba6-ab49-91aa966992b7           
2025-03-10 18:25:51 UTC: Added context url https://friedcheese.us/notice/Arv4zBVnAR84mmkVay                              
2025-03-10 18:25:51 UTC: Error parsing toot URL https://friedcheese.us/notice/Arv4zBVnAR84mmkVay                         
2025-03-10 18:25:51 UTC: Added context url https://friedcheese.us/notice/Arv4pb4zFik9BddiLY                              
2025-03-10 18:25:51 UTC: Error parsing toot URL https://friedcheese.us/notice/Arv4pb4zFik9BddiLY                         
```

I presume that is rate limiting or something at the other end?@nanos: Yeah, I was just looking at this. Has Pleroma changed anything?

Usually Pleroma URLs are of the format `https://{server_name}/objects/{uuid}` as seen in the first url on that list. 

The one it's stumbling over is `https://{server_name}/notice/{something}`. This is weird, because it's at the same domain.

You mentioned that you are running Pleroma. Would you happen to know what's the difference between a `notice` and an `object`?@ltning: Looks like 'object' can refer to various types of items, and in this case the 'objects' URL redirects to the corresponding 'notice'. So notice is a subclass of object, effectively? I don't *know* this, but it would seem logical, to some degree. Why both show up in this particular context is beyond me, though.@nanos: @ltning Have a look at PR #197 

In a nutshell, I believe this is indeed because you are using Pleroma: When we on Mastodon read the timeline we get the `/objects/{id}` url for a Pleroma post, which - as you rightly say - then redirects to `/notice/{id}` when queried. OTOH, when you read the timeline on Pleroma you apparently get a `/notice/{id}` url directly.

Although I don't officially support Pleroma, the change in #197 is straightforward and shouldn't break anything, so if that fixes it, I'm happy to merge it in order to improve things for Pleroma users ...

Are you able to test this for me?

Thanks@ltning: Tested this, here's the output of a single run (note also parsing errors for posts from Hubzilla instances):

```
2025-03-11 10:20:25 UTC: Starting FediFetcher v7.1.15
2025-03-11 10:20:27 UTC: Getting robots.txt file from cache for https://pleroma.anduin.net/robots.txt.
2025-03-11 10:20:27 UTC: Getting context for 0 lists
2025-03-11 10:20:27 UTC: Getting context for home timeline
2025-03-11 10:20:28 UTC: Found 200 toots in timeline
2025-03-11 10:20:28 UTC: Getting robots.txt file from cache for https://mastodon.bsd.cafe/robots.txt.
2025-03-11 10:20:28 UTC: Got context for toot https://mastodon.bsd.cafe/@stefano/114143282658545355
2025-03-11 10:20:28 UTC: Getting robots.txt file from cache for https://mastodon.social/robots.txt.
2025-03-11 10:20:28 UTC: Got context for toot https://mastodon.social/@nixCraft/114143264212808394
2025-03-11 10:20:28 UTC: Getting robots.txt file from cache for https://infosec.exchange/robots.txt.
2025-03-11 10:20:28 UTC: Got context for toot https://infosec.exchange/@SecurityWriter/114143252480542871
2025-03-11 10:20:28 UTC: Getting robots.txt file from cache for https://mastodon.gamedev.place/robots.txt.
2025-03-11 10:20:29 UTC: Got context for toot https://mastodon.gamedev.place/@lritter/114143249137477026
2025-03-11 10:20:29 UTC: Got context for toot https://mastodon.gamedev.place/@lritter/114143246170776119
2025-03-11 10:20:29 UTC: Got context for toot https://pleroma.anduin.net/notice/ArwQy4O9c2JJpkviHw
2025-03-11 10:20:29 UTC: Got context for toot https://mastodon.social/@dec_hl/114143151845612512
2025-03-11 10:20:29 UTC: Got context for toot https://mastodon.social/@po3mah/114143149840039176
2025-03-11 10:20:29 UTC: Got context for toot https://mastodon.gamedev.place/@lritter/114143145447274505
2025-03-11 10:20:29 UTC: Got context for toot https://mastodon.social/@po3mah/114143145142226189
2025-03-11 10:20:29 UTC: Got context for toot https://mastodon.social/@po3mah/114143132432168471
2025-03-11 10:20:30 UTC: Got context for toot https://mastodon.social/@po3mah/114143122181950524
2025-03-11 10:20:30 UTC: Getting robots.txt file from cache for https://cyberplace.social/robots.txt.
2025-03-11 10:20:30 UTC: Got context for toot https://cyberplace.social/@GossiTheDog/114143092762383030
2025-03-11 10:20:30 UTC: Error parsing toot URL https://comam.es/snac/grunfink/p/1741682942.582427
2025-03-11 10:20:30 UTC: Error parsing toot URL https://hub.volse.no/item/623a9e59-0eac-4a96-aec0-59663352a08c
2025-03-11 10:20:30 UTC: Error parsing toot URL https://comam.es/snac/grunfink/p/1741674669.342984
2025-03-11 10:20:30 UTC: Got context for toot https://pleroma.anduin.net/notice/Arvh9kEjlbEuuw9cno
2025-03-11 10:20:30 UTC: Got context for toot https://pleroma.anduin.net/notice/ArvgsV6ykSqv5JIpnM
2025-03-11 10:20:30 UTC: Cannot fetch context of private toot https://infosec.exchange/users/SecurityWriter/statuses/114140612377067810
2025-03-11 10:20:30 UTC: Found 20 known context toots
2025-03-11 10:20:30 UTC: Added 0 new context toots (with 0 failures)
2025-03-11 10:20:30 UTC: Getting robots.txt file from cache for https://bsd.network/robots.txt.
2025-03-11 10:20:30 UTC: Error getting host node info for bsd.network. Exception: Querying https://bsd.network/.well-known/nodeinfo prohibited by robots.txt
2025-03-11 10:20:30 UTC: server bsd.network not found for post
2025-03-11 10:20:30 UTC: Getting robots.txt file from cache for https://comam.es/robots.txt.
2025-03-11 10:20:30 UTC: Error getting host node info for comam.es. Exception: Querying https://comam.es/.well-known/nodeinfo prohibited by robots.txt
2025-03-11 10:20:30 UTC: server comam.es not found for post
2025-03-11 10:20:30 UTC: Getting robots.txt file from cache for https://hub.volse.no/robots.txt.
2025-03-11 10:20:30 UTC: Error getting host node info for hub.volse.no. Exception: Querying https://hub.volse.no/.well-known/nodeinfo prohibited by robots.txt
2025-03-11 10:20:30 UTC: server hub.volse.no not found for post
2025-03-11 10:20:30 UTC: Getting robots.txt file from cache for https://elsewhere.cozysumo.space/robots.txt.
2025-03-11 10:20:30 UTC: Error getting host node info for elsewhere.cozysumo.space. Exception: Querying https://elsewhere.cozysumo.space/.well-known/nodeinfo prohibited by robots.txt
2025-03-11 10:20:30 UTC: server elsewhere.cozysumo.space not found for post
2025-03-11 10:20:30 UTC: server bsd.network not found for post
2025-03-11 10:20:30 UTC: server bsd.network not found for post
2025-03-11 10:20:30 UTC: server bsd.network not found for post
2025-03-11 10:20:30 UTC: server bsd.network not found for post
2025-03-11 10:20:30 UTC: Getting robots.txt file from cache for https://neuro.cooking/robots.txt.
2025-03-11 10:20:30 UTC: Error getting host node info for neuro.cooking. Exception: Querying https://neuro.cooking/.well-known/nodeinfo prohibited by robots.txt
2025-03-11 10:20:30 UTC: server neuro.cooking not found for post
2025-03-11 10:20:30 UTC: Getting posts from last 80 followings
2025-03-11 10:20:30 UTC: Got 80 followings, 19 of which are new
2025-03-11 10:20:30 UTC: Getting robots.txt file from cache for https://larry.weirdr.net/robots.txt.
2025-03-11 10:20:30 UTC: Error getting host node info for larry.weirdr.net. Exception: Querying https://larry.weirdr.net/.well-known/nodeinfo prohibited by robots.txt
2025-03-11 10:20:30 UTC: server larry.weirdr.net not found for post
2025-03-11 10:20:30 UTC: server api unknown for weirdr.net, cannot fetch user posts
2025-03-11 10:20:30 UTC: Error getting host node info for thames.blisses.org. Exception: HTTPSConnectionPool(host='thames.blisses.org', port=443): Max retries exceeded with url: /.well-known/nodeinfo (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x2e2e62dd9250>: Failed to establish a new connection: [Errno 8] Name does not resolve'))
2025-03-11 10:20:30 UTC: server thames.blisses.org not found for post
2025-03-11 10:20:30 UTC: Error getting host node info for slightly.uncomfortable.business. Exception: HTTPSConnectionPool(host='slightly.uncomfortable.business', port=443): Max retries exceeded with url: /.well-known/nodeinfo (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x2e2e62dd99d0>: Failed to establish a new connection: [Errno 1] Address family for hostname not supported'))
2025-03-11 10:20:30 UTC: server slightly.uncomfortable.business not found for post
2025-03-11 10:20:30 UTC: server bsd.network not found for post
2025-03-11 10:20:30 UTC: Getting robots.txt file from cache for https://pix.anduin.net/robots.txt.
2025-03-11 10:20:30 UTC: Error getting user ID for user sndr: Expecting value: line 1 column 1 (char 0)
2025-03-11 10:20:30 UTC: User [bendik@mastodon.bendik.io](mailto:bendik@mastodon.bendik.io) has opted out of backfilling
2025-03-11 10:20:30 UTC: Getting robots.txt file from cache for https://exquisite.social/robots.txt.
2025-03-11 10:20:30 UTC: Error getting host node info for exquisite.social. Exception: Querying https://exquisite.social/.well-known/nodeinfo prohibited by robots.txt
2025-03-11 10:20:30 UTC: server exquisite.social not found for post
2025-03-11 10:20:30 UTC: Error getting host node info for expressional.social. Exception: Querying https://expressional.social/.well-known/nodeinfo prohibited by robots.txt
2025-03-11 10:20:30 UTC: server expressional.social not found for post
2025-03-11 10:20:30 UTC: Error getting user ID for user Vision-Logic: Expecting value: line 1 column 1 (char 0)
2025-03-11 10:20:30 UTC: server comam.es not found for post
2025-03-11 10:20:31 UTC: Error getting user ID for user olgamatna: Expecting value: line 1 column 1 (char 0)
2025-03-11 10:20:31 UTC: Error getting user ID for user 73pctGeek: Expecting value: line 1 column 1 (char 0)
2025-03-11 10:20:31 UTC: server bsd.network not found for post
2025-03-11 10:20:31 UTC: Error getting user ID for user stein: Expecting value: line 1 column 1 (char 0)
2025-03-11 10:20:31 UTC: Getting robots.txt file from cache for https://freaklab.es/robots.txt.
2025-03-11 10:20:31 UTC: Error getting host node info for freaklab.es. Exception: Querying https://freaklab.es/.well-known/nodeinfo prohibited by robots.txt
2025-03-11 10:20:31 UTC: server freaklab.es not found for post
2025-03-11 10:20:31 UTC: Getting robots.txt file from cache for https://pl.pyrope.net/robots.txt.
2025-03-11 10:20:32 UTC: nodeinfo for pl.pyrope.net not found, checking host-meta
2025-03-11 10:20:32 UTC: Error getting host meta for pl.pyrope.net. Status Code: 502
2025-03-11 10:20:32 UTC: server pl.pyrope.net not found for post
2025-03-11 10:20:32 UTC: Getting posts from last 80 followers
2025-03-11 10:20:32 UTC: Got 80 followers, 17 of which are new
2025-03-11 10:20:32 UTC: server api unknown for gts.sfl.pro.br, cannot fetch user posts
2025-03-11 10:20:32 UTC: server api unknown for weirdr.net, cannot fetch user posts
2025-03-11 10:20:32 UTC: server bsd.network not found for post
2025-03-11 10:20:32 UTC: Getting robots.txt file from cache for https://shitposter.world/robots.txt.
2025-03-11 10:20:32 UTC: Error getting host node info for shitposter.world. Exception: Querying https://shitposter.world/.well-known/nodeinfo prohibited by robots.txt
2025-03-11 10:20:32 UTC: server shitposter.world not found for post
2025-03-11 10:20:32 UTC: Error getting host node info for somewhat.uncomfortable.business. Exception: HTTPSConnectionPool(host='somewhat.uncomfortable.business', port=443): Max retries exceeded with url: /.well-known/nodeinfo (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x2e2e62dbf890>: Failed to establish a new connection: [Errno 1] Address family for hostname not supported'))
2025-03-11 10:20:32 UTC: server somewhat.uncomfortable.business not found for post
2025-03-11 10:20:32 UTC: Getting robots.txt file from cache for https://flipboard.com/robots.txt.
2025-03-11 10:20:32 UTC: Error getting host node info for flipboard.com. Exception: Querying https://flipboard.com/.well-known/nodeinfo prohibited by robots.txt
2025-03-11 10:20:32 UTC: server flipboard.com not found for post
2025-03-11 10:20:32 UTC: server slightly.uncomfortable.business not found for post
2025-03-11 10:20:32 UTC: server bsd.network not found for post
2025-03-11 10:20:32 UTC: Getting robots.txt file from cache for https://bencarneiro.com/robots.txt.
2025-03-11 10:20:33 UTC: nodeinfo for bencarneiro.com not found, checking host-meta
2025-03-11 10:20:33 UTC: Error getting host meta for bencarneiro.com. Status Code: 404
2025-03-11 10:20:33 UTC: server bencarneiro.com not found for post
2025-03-11 10:20:33 UTC: Error getting host node info for hub.squiggly.club. Exception: HTTPSConnectionPool(host='hub.squiggly.club', port=443): Max retries exceeded with url: /.well-known/nodeinfo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1006)')))
2025-03-11 10:20:33 UTC: server hub.squiggly.club not found for post
2025-03-11 10:20:33 UTC: server exquisite.social not found for post
2025-03-11 10:20:33 UTC: Getting robots.txt file from cache for https://social.shangtai.net/robots.txt.
2025-03-11 10:20:33 UTC: Error getting host node info for social.shangtai.net. Exception: Querying https://social.shangtai.net/.well-known/nodeinfo prohibited by robots.txt
2025-03-11 10:20:33 UTC: server social.shangtai.net not found for post
2025-03-11 10:20:33 UTC: server api unknown for gts.passthejoe.net, cannot fetch user posts
2025-03-11 10:20:33 UTC: Error getting user ID for user Rampage: Expecting value: line 1 column 1 (char 0)
2025-03-11 10:20:34 UTC: Error getting user ID for user pixel8bit: Expecting value: line 1 column 1 (char 0)
2025-03-11 10:20:34 UTC: Getting posts from last 80 follow requests
2025-03-11 10:20:34 UTC: Got 0 follow_requests, 0 of which are new
2025-03-11 10:20:34 UTC: Getting notifications for last 3 hours
2025-03-11 10:20:34 UTC: Found 0 users in notifications, 0 of which are new
2025-03-11 10:20:34 UTC: Pulling replies to the last 80 bookmarks
2025-03-11 10:20:34 UTC: Found 0 known context toots
2025-03-11 10:20:34 UTC: Added 0 new context toots (with 0 failures)
2025-03-11 10:20:34 UTC: Pulling replies to the last 40 favourites
2025-03-11 10:20:34 UTC: Discovered redirect for URL https://soc.punktrash.club/objects/c26f20a1-b6d5-4817-bc60-fd359d56f0d0
2025-03-11 10:20:34 UTC: Discovered redirect for URL https://fed.xnor.in/objects/6fe457ec-138a-471b-aca3-5a83f6f442f5
2025-03-11 10:20:34 UTC: Error parsing toot URL https://hub.volse.no/item/ab4d80f0-9416-4543-9ab9-f6d4990447ef
2025-03-11 10:20:34 UTC: Error parsing toot URL https://hub.volse.no/item/01fb35b8-c163-493b-9c83-1ad6b747faa9
2025-03-11 10:20:34 UTC: Cannot fetch context of private toot https://fribygda.no/users/amici/statuses/114029909723281600
2025-03-11 10:20:34 UTC: Found 0 known context toots
2025-03-11 10:20:34 UTC: Added 0 new context toots (with 0 failures)
2025-03-11 10:20:35 UTC: Processing finished in 0:00:09.975154.
```
@nanos: nice one! this seems to be working as far as I can see.

Solution patch:
diff --git a/find_posts.py b/find_posts.py
index 82cd002a..2ddfbce5 100644
--- a/find_posts.py
+++ b/find_posts.py
@@ -628,6 +628,11 @@ def parse_url(url, parsed_urls):
         if match is not None:
             parsed_urls[url] = match
 
+    if url not in parsed_urls:
+        match = parse_pleroma_uri(url)
+        if match is not None:
+            parsed_urls[url] = match
+
     if url not in parsed_urls:
         match = parse_lemmy_url(url)
         if match is not None:
@@ -696,6 +701,13 @@ def parse_pleroma_url(url):
         return None
     return None
 
+def parse_pleroma_uri(uri):
+    """parse a Pleroma URL and return the server and ID"""
+    match = re.match(r"https://(?P<server>[^/]+)/notice/(?P<toot_id>[^/]+)", uri)
+    if match is not None:
+        return (match.group("server"), match.group("toot_id"))
+    return None
+
 def parse_pleroma_profile_url(url):
     """parse a Pleroma Profile URL and return the server and username"""
     match = re.match(r"https://(?P<server>[^/]+)/users/(?P<username>[^/]+)", url)


Test patch:
diff --git a/tests/test_find_posts.py b/tests/test_find_posts.py
index 6e633d53..7a4fa45d 100644
--- a/tests/test_find_posts.py
+++ b/tests/test_find_posts.py
@@ -39,6 +39,7 @@
     parse_pixelfed_profile_url,
     parse_pixelfed_url,
     parse_pleroma_url,
+    parse_pleroma_uri,
     post,
     set_server_apis,
     user_has_opted_out,
@@ -874,6 +875,10 @@ def test_parse_pleroma_url(mock_get_redirect_url):
     result = parse_pleroma_url("https://different.example.com/objects/111")
     assert result == ("different.example.com", "789")
 
+def test_parse_pleroma_uri():
+    # Test that a valid URI is correctly parsed
+    uri = "https://friedcheese.us/notice/Arv4zBVnAR84mmkVay"
+    assert parse_pleroma_uri(uri) == ("friedcheese.us", "Arv4zBVnAR84mmkVay")    
 
 import re
 import pytest