Repository: wagtail/wagtail

Issue:
Caching OEmbed markup prevents markup changes for existing embeds
The database caching strategy keeps the rendered embed markup. This means that if you implement a change like https://gist.github.com/kaedroho/ec2b2e83f2f4ecfb9677 to fix an issue with an embed provider, any existing embeds don't get fixed.

My thinking would be to cache the output of the oembed request instead of the embed itself, then re-render the embed from that on subsequent lookups.

Caching OEmbed markup prevents markup changes for existing embeds
The database caching strategy keeps the rendered embed markup. This means that if you implement a change like https://gist.github.com/kaedroho/ec2b2e83f2f4ecfb9677 to fix an issue with an embed provider, any existing embeds don't get fixed.

My thinking would be to cache the output of the oembed request instead of the embed itself, then re-render the embed from that on subsequent lookups.



Discussion:
Would like to chime in that this issue really bit us in production, as a change to the embed functions (to allow https in YouTube embeds) was not reflected in previously embedded videos, causing quite a bit of confusion and scrambling during a deadline. At the end of the day we had to modify the embed markup directly in the DB to solve the issue.

Thanks for reporting this. A quick fix would be to clear out the `Embed` model every time you make a change to your embeds settings (this will cause all embeds to be refetched on the next request).

I don't understand.

@SableWalnut I think what @kaedroho meant is to delete all `Embed `-objects:

`python manage.py shell`

```python
from wagtail.embeds.models import Embed
Embed.objects.all().delete()
```

After that, wagtail will lookup the embed code again.
Older issue but a potential candidate for a management command? 

Flagging as a good first issue to implement this approach based on the code snippet above, documentation could also be included in the embeds for this.

We already flag that you may want to delete embeds if you change something here https://docs.wagtail.org/en/stable/advanced_topics/embeds.html#deleting-embeds

So that would be a good place to add a link to the management command.

See https://github.com/wagtail/wagtail/issues/8107 for a similar management command suggested approach for image renditions.
may i take this one up ?
@4the4ryushin - yeah go for it but please check you understand the issue fully and can reproduce the problem as described.

It may be good to add a comment with the exact steps to reproduce a scenario like this on a new Wagtail project, this will also help for any kind of PR validation when you prepare it.

@lb-  i have understood the issue would soon write the steps to reproduce it,  so we are approaching the issue in way that 
we need to write a management command for deleting the embeds in order to wagtail to re fetch embed code?
what should i name the command?
Have a look at existing commands, we would need docs also. 

https://docs.wagtail.org/en/stable/reference/management_commands.html

Probably `wagtail_purge_embeds` or just `purge_embeds`
Got it, on it 
Here is the code and the tests for the purge revisions command, the new command can be implemented very much the same. Without needing any Params support. 

https://github.com/wagtail/wagtail/blob/main/wagtail/management/commands/purge_revisions.py

Tests. 
https://github.com/wagtail/wagtail/blob/main/wagtail/tests/test_management_commands.py#L591

However, these files need to live in the embeds app folder
https://github.com/wagtail/wagtail/tree/main/wagtail/embeds
@lb-  thank you so much, very much appreciated saved me a lot of " dig up "
> The database caching strategy keeps the rendered embed markup. This means that if you implement a change like https://gist.github.com/kaedroho/ec2b2e83f2f4ecfb9677 to fix an issue with an embed provider, any existing embeds don't get fixed.
> 
> My thinking would be to cache the output of the oembed request instead of the embed itself, then re-render the embed from that on subsequent lookups.

apparently the file for custom embed_finder code imports oembed function which seems like it doesn't exist anymore so i am making a new way to reproduce the error.
> The database caching strategy keeps the rendered embed markup. This means that if you implement a change like https://gist.github.com/kaedroho/ec2b2e83f2f4ecfb9677 to fix an issue with an embed provider, any existing embeds don't get fixed.
> 
> My thinking would be to cache the output of the oembed request instead of the embed itself, then re-render the embed from that on subsequent lookups.

STEPS TO REPRODUCE THE ISSUE:

1.  create  a normal blog page , you can even follow  getting started guide https://docs.wagtail.org/en/stable/getting_started/tutorial.html#

2.  In any of your template  load embed tags and and use the tag anywhere in the template to embed a demo video
 use the embed tag, https://docs.wagtail.org/en/stable/advanced_topics/embeds.html#embed-tag
 
3.  in your settings.py put the following code, 
```python
WAGTAILEMBEDS_FINDERS = [
    {
        'class': 'wagtail.embeds.finders.oembed'
    }
]
```
4.   Create a file in your app folder eg :- myapp/embeds.py
5.   put the following code in embeds.py to create custom embed finder,  https://gist.github.com/4the4ryushin/e6a5b3c0a588883538ca5fb7b5ed4e12 
at line 50 i have hardcoded an another video url to observe the changes brought by it so this custom finder would always embeds the same video regardless of any url.

6.  change the settings to use the custom embed finder 
 ```python
WAGTAILEMBEDS_FINDERS = [
    {
        'class': 'myapp.embeds.CustomEmbedFinder'
    }
]
```
7. now run the server and reload the blog page to see  if the embed video changes to our hard coded video 

expected behaviour : video  should have changed to hard coded video, as we replaced our new custom embed finder
actual behaviour:  video doesn't changes unless we deleted the Embed model object and refetch the code again .
this addresses the issue #1267

I have verified that this issue can be reproduced on latest wagtail projects 
python : 3.10.4
wagtail : 4.0
Would like to chime in that this issue really bit us in production, as a change to the embed functions (to allow https in YouTube embeds) was not reflected in previously embedded videos, causing quite a bit of confusion and scrambling during a deadline. At the end of the day we had to modify the embed markup directly in the DB to solve the issue.

Thanks for reporting this. A quick fix would be to clear out the `Embed` model every time you make a change to your embeds settings (this will cause all embeds to be refetched on the next request).

I don't understand.

@SableWalnut I think what @kaedroho meant is to delete all `Embed `-objects:

`python manage.py shell`

```python
from wagtail.embeds.models import Embed
Embed.objects.all().delete()
```

After that, wagtail will lookup the embed code again.
Older issue but a potential candidate for a management command? 

Flagging as a good first issue to implement this approach based on the code snippet above, documentation could also be included in the embeds for this.

We already flag that you may want to delete embeds if you change something here https://docs.wagtail.org/en/stable/advanced_topics/embeds.html#deleting-embeds

So that would be a good place to add a link to the management command.

See https://github.com/wagtail/wagtail/issues/8107 for a similar management command suggested approach for image renditions.
may i take this one up ?
@4the4ryushin - yeah go for it but please check you understand the issue fully and can reproduce the problem as described.

It may be good to add a comment with the exact steps to reproduce a scenario like this on a new Wagtail project, this will also help for any kind of PR validation when you prepare it.

@lb-  i have understood the issue would soon write the steps to reproduce it,  so we are approaching the issue in way that 
we need to write a management command for deleting the embeds in order to wagtail to re fetch embed code?
what should i name the command?
Have a look at existing commands, we would need docs also. 

https://docs.wagtail.org/en/stable/reference/management_commands.html

Probably `wagtail_purge_embeds` or just `purge_embeds`
Got it, on it 
Here is the code and the tests for the purge revisions command, the new command can be implemented very much the same. Without needing any Params support. 

https://github.com/wagtail/wagtail/blob/main/wagtail/management/commands/purge_revisions.py

Tests. 
https://github.com/wagtail/wagtail/blob/main/wagtail/tests/test_management_commands.py#L591

However, these files need to live in the embeds app folder
https://github.com/wagtail/wagtail/tree/main/wagtail/embeds
@lb-  thank you so much, very much appreciated saved me a lot of " dig up "
> The database caching strategy keeps the rendered embed markup. This means that if you implement a change like https://gist.github.com/kaedroho/ec2b2e83f2f4ecfb9677 to fix an issue with an embed provider, any existing embeds don't get fixed.
> 
> My thinking would be to cache the output of the oembed request instead of the embed itself, then re-render the embed from that on subsequent lookups.

apparently the file for custom embed_finder code imports oembed function which seems like it doesn't exist anymore so i am making a new way to reproduce the error.
> The database caching strategy keeps the rendered embed markup. This means that if you implement a change like https://gist.github.com/kaedroho/ec2b2e83f2f4ecfb9677 to fix an issue with an embed provider, any existing embeds don't get fixed.
> 
> My thinking would be to cache the output of the oembed request instead of the embed itself, then re-render the embed from that on subsequent lookups.

STEPS TO REPRODUCE THE ISSUE:

1.  create  a normal blog page , you can even follow  getting started guide https://docs.wagtail.org/en/stable/getting_started/tutorial.html#

2.  In any of your template  load embed tags and and use the tag anywhere in the template to embed a demo video
 use the embed tag, https://docs.wagtail.org/en/stable/advanced_topics/embeds.html#embed-tag
 
3.  in your settings.py put the following code, 
```python
WAGTAILEMBEDS_FINDERS = [
    {
        'class': 'wagtail.embeds.finders.oembed'
    }
]
```
4.   Create a file in your app folder eg :- myapp/embeds.py
5.   put the following code in embeds.py to create custom embed finder,  https://gist.github.com/4the4ryushin/e6a5b3c0a588883538ca5fb7b5ed4e12 
at line 50 i have hardcoded an another video url to observe the changes brought by it so this custom finder would always embeds the same video regardless of any url.

6.  change the settings to use the custom embed finder 
 ```python
WAGTAILEMBEDS_FINDERS = [
    {
        'class': 'myapp.embeds.CustomEmbedFinder'
    }
]
```
7. now run the server and reload the blog page to see  if the embed video changes to our hard coded video 

expected behaviour : video  should have changed to hard coded video, as we replaced our new custom embed finder
actual behaviour:  video doesn't changes unless we deleted the Embed model object and refetch the code again .
this addresses the issue #1267

I have verified that this issue can be reproduced on latest wagtail projects 
python : 3.10.4
wagtail : 4.0

Resolving patch:
diff --git a/wagtail/management/commands/purge_embeds.py b/wagtail/management/commands/purge_embeds.py
new file mode 100644
--- /dev/null
+++ b/wagtail/management/commands/purge_embeds.py
@@ -0,0 +1,21 @@
+from django.core.management.base import BaseCommand
+
+from wagtail.embeds.models import Embed
+
+
+class Command(BaseCommand):
+    help = "Deletes all of the Embed model objects"
+
+    def handle(self, *args, **options):
+
+        embeds = Embed.objects.all()
+
+        deleted_embeds_count = embeds.delete()[0]
+        if deleted_embeds_count:
+            self.stdout.write(
+                self.style.SUCCESS(
+                    f"Successfully deleted {deleted_embeds_count} embeds"
+                )
+            )
+        else:
+            self.stdout.write("Successfully deleted 0 embeds")


Test patch:
diff --git a/wagtail/tests/test_management_commands.py b/wagtail/tests/test_management_commands.py
--- a/wagtail/tests/test_management_commands.py
+++ b/wagtail/tests/test_management_commands.py
@@ -9,6 +9,7 @@
 from django.test import TestCase
 from django.utils import timezone
 
+from wagtail.embeds.models import Embed
 from wagtail.models import Collection, Page, PageLogEntry, Revision
 from wagtail.signals import page_published, page_unpublished, published, unpublished
 from wagtail.test.testapp.models import (
@@ -697,6 +698,40 @@ def test_purge_revisions_with_date_cutoff(self):
         )
 
 
+class TestPurgeEmbedsCommand(TestCase):
+    fixtures = ["test.json"]
+
+    def setUp(self):
+        # create dummy Embed objects
+        for i in range(5):
+            embed = Embed(
+                hash=f"{i}",
+                url="https://www.youtube.com/watch?v=Js8dIRxwSRY",
+                max_width=None,
+                type="video",
+                html="test html",
+                title="test title",
+                author_name="test author name",
+                provider_name="test provider name",
+                thumbnail_url="http://test/thumbnail.url",
+                width=1000,
+                height=1000,
+            )
+            embed.save()
+
+    def test_purge_embeds(self):
+        """
+        fetch all dummy embeds and confirm they are deleted when the management command runs
+
+        """
+
+        self.assertEqual(Embed.objects.count(), 5)
+
+        management.call_command("purge_embeds", stdout=StringIO())
+
+        self.assertEqual(Embed.objects.count(), 0)
+
+
 class TestCreateLogEntriesFromRevisionsCommand(TestCase):
     fixtures = ["test.json"]