Upload 124 files

.gitattributes

@@ -32,3 +32,9 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+sd-webui-cutoff/images/cover.jpg filter=lfs diff=lfs merge=lfs -text
+sd-webui-cutoff/images/sample-1.png filter=lfs diff=lfs merge=lfs -text
+sd-webui-cutoff/images/sample-2.png filter=lfs diff=lfs merge=lfs -text
+sd-webui-cutoff/images/sample-3.png filter=lfs diff=lfs merge=lfs -text
+sd-webui-llul/images/llul_yuv420p.mp4 filter=lfs diff=lfs merge=lfs -text
+sd-webui-llul/images/mask_effect.jpg filter=lfs diff=lfs merge=lfs -text

infinite-zoom-automatic1111-webui/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,29 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: "[BUG]"
+labels: bug
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Enter these inputs '....'
+2. Click on '....'
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots of error**
+If applicable, add screenshots to help explain your problem. Or just dump the error.
+
+**Desktop (please complete the following information):**
+ - OS: [e.g. Windows]
+ - Version [e.g. 22]
+
+**Additional context**
+Add any other context about the problem here.

infinite-zoom-automatic1111-webui/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: "[FEATURE]"
+labels: enhancement
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

infinite-zoom-automatic1111-webui/.gitignore

@@ -0,0 +1,132 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+.vscode/settings.json
+.DS_Store
+/.vs

infinite-zoom-automatic1111-webui/CODE_OF_CONDUCT.md

@@ -0,0 +1,128 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, religion, or sexual identity
+and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the
+  overall community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or
+  advances of any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email
+  address, without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official e-mail address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+me@vahid.cloud.
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series
+of actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or
+permanent ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior,  harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within
+the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.0, available at
+https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
+
+Community Impact Guidelines were inspired by [Mozilla's code of conduct
+enforcement ladder](https://github.com/mozilla/diversity).
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see the FAQ at
+https://www.contributor-covenant.org/faq. Translations are available at
+https://www.contributor-covenant.org/translations.

infinite-zoom-automatic1111-webui/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 vahid khroasani
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

infinite-zoom-automatic1111-webui/README.md

@@ -0,0 +1,115 @@
+# Infinite Zoom extension for  [Stable Diffusion WebUI](https://github.com/AUTOMATIC1111/stable-diffusion-webui/).  
+
+<p align="center">     
+    <a href="https://discord.gg/v2nHqSrWdW">
+        <img src="https://img.shields.io/discord/1095469311830806630?color=blue&label=discord&logo=discord&logoColor=white" alt="build status">
+    </a>
+</p>
+
+This is an extension for the AUTOMATIC1111's (and Vladmandic´s) webui that allows users to create infinite zoom effect videos using stable diffusion outpainting method. 
+<p align="center">
+   <img src="https://user-images.githubusercontent.com/62482657/233385585-82d7157e-1438-4cf8-b805-220d96bbbe31.gif" width="332" height="188" />
+</p>
+
+## How to install?
+<details>
+  <summary> Click to expand </summary>
+  
+1. Open [Stable Diffusion WebUI](https://github.com/AUTOMATIC1111/stable-diffusion-webui/).
+
+2. Go to the `Extensions tab` > `Install from URL`.
+
+3. Enter `https://github.com/v8hid/infinite-zoom-automatic1111-webui.git` for the URL and leave the second field empty and wait for it to be installed.
+> <img width="587" alt="Screenshot" src="https://user-images.githubusercontent.com/62482657/231554653-16484c48-582e-489d-8191-bafc4cccbd3b.png">
+
+4. Go to the Installed tab and press Apply, wait for installation, and restart.
+
+> <img width="616" alt="Screenshot" src="https://user-images.githubusercontent.com/62482657/231554793-4a54ae94-51d2-408e-9908-2eed73cde9c0.png">
+
+5. Wait for the Stable Diffusion WebUI to restart and now you can try the Infinite Zoom extension.
+
+</details>
+
+## How to use?
+
+<details>
+  <summary> Click to expand </summary>
+  
+ 1. Click on the Infinite Zoom tab <img width="1431" alt="Screenshot 2023-04-12 at 10 14 50 PM" src="https://user-images.githubusercontent.com/62482657/231571341-92767f0d-af36-4b94-8ba9-c40a63c209ba.png">
+ 
+ 2. Modify the parameters as you wish and click Generate video, the video will appear as soon as it generates
+ 
+ </details>
+ 
+**To learn more about the parameters, please refer to our [WIKI](https://github.com/v8hid/infinite-zoom-automatic1111-webui/wiki).**
+ ## Effective Friendly Tips for Optimal Outcomes
+ 
+<details>
+  <summary> Click to expand </summary>
+  
+* You're only as good as your model, so level up with an <ins>Inpainting model</ins> for killer results.
+
+* Heads up: Setting <ins>Mask Blur</ins> parameter above 0 will give you results that look like they've been hit by the ugly stick.
+
+* Just between us - don't forget to uncheck <ins> Apply color correction to img2img results to match original colors</ins> in the Stable Diffusion tab of the WebUI settings. You don't want your results looking like a bad Instagram filter.
+
+</details>
+
+## Examples
+
+<details>
+  <summary> Click to expand </summary>
+
+
+
+https://user-images.githubusercontent.com/62482657/232369614-e112d17a-db12-47b2-9795-5be4037fa9fe.mp4
+
+
+https://user-images.githubusercontent.com/62482657/231573289-2db85c57-540d-4c7d-859f-3c3ddfcd2c8a.mp4
+
+
+https://user-images.githubusercontent.com/62482657/231574588-3196beda-7237-407f-bc76-eae10599b5eb.mp4
+
+
+https://user-images.githubusercontent.com/62482657/231574839-9d3aab52-7a87-4658-88d0-46b8dd7f4b60.mp4
+
+ </details>
+
+## How it works?
+<details>
+  <summary> Click to expand </summary>
+  
+To start, let's break down the workflow of the extension into three main steps:
+
+- **Step 1: Choose an image to start with**
+The program either generates an initial image using the first prompt you provide or you can upload your own image in the `custom initial image` field. This initial image will be the basis for the outpainting process.
+
+- **Step 2: Generate outpaint steps**
+Once you have your initial image, the program will start generating outpaint steps. The number of outpaint steps is determined by the `Total Outpaint Steps` input. In each outpaint step, the program makes the initial image smaller in the center of the canvas and generates a new image in the empty space that is created. This process is repeated for each outpaint step until the desired number is reached.
+
+- **Step 3: Create a gradual zoom effect**
+After all outpaint steps have been generated, the program creates an interpolation between each outpaint step to create a gradual zoom effect. The number of frames created between each outpaint step is determined by the `Zoom Speed` parameter and the `Frames per second` parameter.
+
+Number of frames for each outpaint step = `Zoom Speed` $\times$ `Frames per second`
+
+Length of each outpaint step in second = `Number of frames` $\div$ `Frames per second` 
+
+ </details>
+ 
+## Google Colab version
+It works on free colab plan
+
+<a target="_blank" href="https://colab.research.google.com/github/v8hid/infinite-zoom-stable-diffusion/blob/main/infinite_zoom_gradio.ipynb">
+  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
+</a>
+<a target="_blank" href="https://github.com/v8hid/infinite-zoom-stable-diffusion">
+  <img src="https://img.shields.io/static/v1?label=github&message=repository&color=blue&style=flat&logo=github&logoColor=white" alt="GitHub Repo"/>
+</a>
+
+## Contributing
+
+Contributions are welcome! Please follow these guidelines:
+
+  1. Fork the repository.
+  2. Make your changes and commit them.
+  3. Make sure to submit the pull request to the develop repository.

infinite-zoom-automatic1111-webui/install.py

@@ -0,0 +1,6 @@
+import launch
+
+if not launch.is_installed("imageio"):
+    launch.run_pip("install imageio", "requirements 0 for Infinite-Zoom")
+if not launch.is_installed("imageio-ffmpeg"):
+    launch.run_pip("install imageio-ffmpeg", "requirements 1 for Infinite-Zoom")

infinite-zoom-automatic1111-webui/iz_helpers/__init__.py

@@ -0,0 +1,2 @@
+# from .ui import on_ui_tabs
+# from .settings import on_ui_settings

infinite-zoom-automatic1111-webui/iz_helpers/helpers.py

@@ -0,0 +1,126 @@
+import math
+import os
+import modules.shared as shared
+import modules.sd_models
+import gradio as gr
+from scripts import postprocessing_upscale
+from .prompt_util import readJsonPrompt
+import asyncio
+
+
+def fix_env_Path_ffprobe():
+    envpath = os.environ["PATH"]
+    ffppath = shared.opts.data.get("infzoom_ffprobepath", "")
+
+    if ffppath and not ffppath in envpath:
+        path_sep = ";" if os.name == "nt" else ":"
+        os.environ["PATH"] = envpath + path_sep + ffppath
+
+
+def closest_upper_divisible_by_eight(num):
+    if num % 8 == 0:
+        return num
+    else:
+        return math.ceil(num / 8) * 8
+
+
+def load_model_from_setting(model_field_name, progress, progress_desc):
+    # fix typo in Automatic1111 vs Vlad111
+    if hasattr(modules.sd_models, "checkpoint_alisases"):
+        checkPList = modules.sd_models.checkpoint_alisases
+    elif hasattr(modules.sd_models, "checkpoint_aliases"):
+        checkPList = modules.sd_models.checkpoint_aliases
+    else:
+        raise Exception(
+            "This is not a compatible StableDiffusion Platform, can not access checkpoints"
+        )
+
+    model_name = shared.opts.data.get(model_field_name)
+    if model_name is not None and model_name != "":
+        checkinfo = checkPList[model_name]
+
+        if not checkinfo:
+            raise NameError(model_field_name + " Does not exist in your models.")
+
+        if progress:
+            progress(0, desc=progress_desc + checkinfo.name)
+
+        modules.sd_models.load_model(checkinfo)
+
+
+def do_upscaleImg(curImg, upscale_do, upscaler_name, upscale_by):
+    if not upscale_do:
+        return curImg
+
+    # ensure even width and even height for ffmpeg
+    # if odd, switch to scale to mode
+    rwidth = round(curImg.width * upscale_by)
+    rheight = round(curImg.height * upscale_by)
+
+    ups_mode = 2  # upscale_by
+    if (rwidth % 2) == 1:
+        ups_mode = 1
+        rwidth += 1
+    if (rheight % 2) == 1:
+        ups_mode = 1
+        rheight += 1
+
+    if 1 == ups_mode:
+        print(
+            "Infinite Zoom: aligning output size to even width and height: "
+            + str(rwidth)
+            + " x "
+            + str(rheight),
+            end="\r",
+        )
+
+    pp = postprocessing_upscale.scripts_postprocessing.PostprocessedImage(curImg)
+    ups = postprocessing_upscale.ScriptPostprocessingUpscale()
+    ups.process(
+        pp,
+        upscale_mode=ups_mode,
+        upscale_by=upscale_by,
+        upscale_to_width=rwidth,
+        upscale_to_height=rheight,
+        upscale_crop=False,
+        upscaler_1_name=upscaler_name,
+        upscaler_2_name=None,
+        upscaler_2_visibility=0.0,
+    )
+    return pp.image
+
+async def showGradioErrorAsync(txt, delay=1):
+    await asyncio.sleep(delay)  # sleep for 1 second
+    raise gr.Error(txt)
+
+def putPrompts(files):
+    try:
+        with open(files.name, "r") as f:
+            file_contents = f.read()
+
+            data = readJsonPrompt(file_contents,False)
+            return [
+                gr.Textbox.update(data["prePrompt"]),
+                gr.DataFrame.update(data["prompts"]),
+                gr.Textbox.update(data["postPrompt"]),
+                gr.Textbox.update(data["negPrompt"])
+            ]
+
+    except Exception:
+        print(
+            "[InfiniteZoom:] Loading your prompt failed. It seems to be invalid. Your prompt table is preserved."
+        )
+        
+        # error only be shown with raise, so ui gets broken.
+        #asyncio.run(showGradioErrorAsync("Loading your prompts failed. It seems to be invalid. Your prompt table has been preserved.",5))
+
+        return [gr.Textbox.update(), gr.DataFrame.update(), gr.Textbox.update(),gr.Textbox.update()]
+
+
+def clearPrompts():
+    return [
+        gr.DataFrame.update(value=[[0, "Infinite Zoom. Start over"]]),
+        gr.Textbox.update(""),
+        gr.Textbox.update(""),
+        gr.Textbox.update("")
+    ]

infinite-zoom-automatic1111-webui/iz_helpers/image.py

@@ -0,0 +1,23 @@
+from PIL import Image
+
+def shrink_and_paste_on_blank(current_image, mask_width, mask_height):
+    """
+    Decreases size of current_image by mask_width pixels from each side,
+    then adds a mask_width width transparent frame,
+    so that the image the function returns is the same size as the input.
+    :param current_image: input image to transform
+    :param mask_width: width in pixels to shrink from each side
+    :param mask_height: height in pixels to shrink from each side
+    """
+
+    # calculate new dimensions
+    width, height = current_image.size
+    new_width = width - 2 * mask_width
+    new_height = height - 2 * mask_height
+
+    # resize and paste onto blank image
+    prev_image = current_image.resize((new_width, new_height))
+    blank_image = Image.new("RGBA", (width, height), (0, 0, 0, 1))
+    blank_image.paste(prev_image, (mask_width, mask_height))
+
+    return blank_image

infinite-zoom-automatic1111-webui/iz_helpers/prompt_util.py

@@ -0,0 +1,67 @@
+import json
+from jsonschema import validate
+
+from .static_variables import (
+    empty_prompt,
+    invalid_prompt,
+    jsonprompt_schemafile,
+    promptTableHeaders
+)
+
+def completeOptionals(j):
+    if isinstance(j, dict):
+        # Remove header information, user dont pimp our ui
+        if "prompts" in j:
+            if "headers" in j["prompts"]:
+                del j["prompts"]["headers"]
+            j["prompts"]["headers"]=promptTableHeaders
+            
+        if "negPrompt" not in j:
+            j["negPrompt"]=""
+            
+        if "prePrompt" not in j:
+            if "commonPromptPrefix" in j:
+                j["prePrompt"]=j["commonPromptPrefix"]
+            else:
+                j["prePrompt"]=""
+            
+        if "postPrompt" not in j:
+            if "commonPromptSuffix" in j:
+                j["postPrompt"]=j["commonPromptSuffix"]
+            else:
+                j["postPrompt"]=""
+
+    return j
+
+
+def validatePromptJson_throws(data):
+    with open(jsonprompt_schemafile, "r") as s:
+        schema = json.load(s)
+    try:
+        validate(instance=data, schema=schema)
+       
+    except Exception:
+        raise Exception("Your prompts are not schema valid.")
+
+    return completeOptionals(data)
+
+
+def readJsonPrompt(txt, returnFailPrompt=False):
+    if not txt:
+        return empty_prompt
+
+    try:
+        jpr = json.loads(txt)
+    except Exception:
+        if returnFailPrompt:
+            print (f"Infinite Zoom: Corrupted Json structure: {txt[:24]} ...")
+            return invalid_prompt
+        raise (f"Infinite Zoom: Corrupted Json structure: {txt[:24]} ...")
+
+    try:
+        return validatePromptJson_throws(jpr)
+    except Exception:
+        if returnFailPrompt:
+            return invalid_prompt
+        pass
+    

infinite-zoom-automatic1111-webui/iz_helpers/promptschema.json

@@ -0,0 +1,60 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "$id": "1.1",
+  "type": "object",
+  "properties": {
+    "prompts": {
+      "type": "object",
+      "properties": {
+        "data": {
+          "type": "array",
+          "items": {
+            "type": "array",
+            "items": [
+              {
+                "oneOf": [
+                  {
+                    "type": "integer",
+                    "minimum": 0
+                  },
+                  {
+                    "type": "string"
+                  }
+                ]
+              },
+              {
+                "type": "string"
+              }
+            ],
+            "minItems": 0,
+            "maxItems": 999,
+            "uniqueItems": false
+          },
+          "minItems": 0
+        },
+        "headers": {
+          "type": "array",
+          "items": {
+            "type": "string"
+          },
+          "minItems": 2
+        }
+      },
+      "required": [
+        "data"
+      ]
+    },
+    "negPrompt": {
+      "type": "string"
+    },
+    "prePrompt": {
+      "type": "string"
+    },
+    "postPrompt": {
+      "type": "string"
+    }
+  },
+  "required": [
+    "prompts"
+  ]
+}

infinite-zoom-automatic1111-webui/iz_helpers/run.py

@@ -0,0 +1,512 @@
+import math, time, os
+import numpy as np
+from PIL import Image, ImageFilter, ImageDraw
+from modules.ui import plaintext_to_html
+import modules.shared as shared
+from modules.paths_internal import script_path
+from .helpers import (
+    fix_env_Path_ffprobe,
+    closest_upper_divisible_by_eight,
+    load_model_from_setting,
+    do_upscaleImg,
+)
+from .sd_helpers import renderImg2Img, renderTxt2Img
+from .image import shrink_and_paste_on_blank
+from .video import write_video
+
+
+def crop_fethear_ellipse(image, feather_margin=30, width_offset=0, height_offset=0):
+    # Create a blank mask image with the same size as the original image
+    mask = Image.new("L", image.size, 0)
+    draw = ImageDraw.Draw(mask)
+
+    # Calculate the ellipse's bounding box
+    ellipse_box = (
+        width_offset,
+        height_offset,
+        image.width - width_offset,
+        image.height - height_offset,
+    )
+
+    # Draw the ellipse on the mask
+    draw.ellipse(ellipse_box, fill=255)
+
+    # Apply the mask to the original image
+    result = Image.new("RGBA", image.size)
+    result.paste(image, mask=mask)
+
+    # Crop the resulting image to the ellipse's bounding box
+    cropped_image = result.crop(ellipse_box)
+
+    # Create a new mask image with a black background (0)
+    mask = Image.new("L", cropped_image.size, 0)
+    draw = ImageDraw.Draw(mask)
+
+    # Draw an ellipse on the mask image
+    draw.ellipse(
+        (
+            0 + feather_margin,
+            0 + feather_margin,
+            cropped_image.width - feather_margin,
+            cropped_image.height - feather_margin,
+        ),
+        fill=255,
+        outline=0,
+    )
+
+    # Apply a Gaussian blur to the mask image
+    mask = mask.filter(ImageFilter.GaussianBlur(radius=feather_margin / 2))
+    cropped_image.putalpha(mask)
+    res = Image.new(cropped_image.mode, (image.width, image.height))
+    paste_pos = (
+        int((res.width - cropped_image.width) / 2),
+        int((res.height - cropped_image.height) / 2),
+    )
+    res.paste(cropped_image, paste_pos)
+
+    return res
+
+
+def outpaint_steps(
+    width,
+    height,
+    common_prompt_pre,
+    common_prompt_suf,
+    prompts,
+    negative_prompt,
+    seed,
+    sampler,
+    num_inference_steps,
+    guidance_scale,
+    inpainting_denoising_strength,
+    inpainting_mask_blur,
+    inpainting_fill_mode,
+    inpainting_full_res,
+    inpainting_padding,
+    init_img,
+    outpaint_steps,
+    out_config,
+    mask_width,
+    mask_height,
+    custom_exit_image,
+    frame_correction=True,  # TODO: add frame_Correction in UI
+):
+    main_frames = [init_img.convert("RGB")]
+
+    for i in range(outpaint_steps):
+        print_out = (
+            "Outpaint step: "
+            + str(i + 1)
+            + " / "
+            + str(outpaint_steps)
+            + " Seed: "
+            + str(seed)
+        )
+        print(print_out)
+        current_image = main_frames[-1]
+        current_image = shrink_and_paste_on_blank(
+            current_image, mask_width, mask_height
+        )
+
+        mask_image = np.array(current_image)[:, :, 3]
+        mask_image = Image.fromarray(255 - mask_image).convert("RGB")
+        # create mask (black image with white mask_width width edges)
+
+        if custom_exit_image and ((i + 1) == outpaint_steps):
+            current_image = custom_exit_image.resize(
+                (width, height), resample=Image.LANCZOS
+            )
+            main_frames.append(current_image.convert("RGB"))
+            # print("using Custom Exit Image")
+            save2Collect(current_image, out_config, f"exit_img.png")
+        else:
+            pr = prompts[max(k for k in prompts.keys() if k <= i)]
+            processed, newseed = renderImg2Img(
+                f"{common_prompt_pre}\n{pr}\n{common_prompt_suf}".strip(),
+                negative_prompt,
+                sampler,
+                num_inference_steps,
+                guidance_scale,
+                seed,
+                width,
+                height,
+                current_image,
+                mask_image,
+                inpainting_denoising_strength,
+                inpainting_mask_blur,
+                inpainting_fill_mode,
+                inpainting_full_res,
+                inpainting_padding,
+            )
+
+            if len(processed.images) > 0:
+                main_frames.append(processed.images[0].convert("RGB"))
+                save2Collect(processed.images[0], out_config, f"outpain_step_{i}.png")
+            seed = newseed
+            # TODO: seed behavior
+
+        if frame_correction and inpainting_mask_blur > 0:
+            corrected_frame = crop_inner_image(
+                main_frames[i + 1], mask_width, mask_height
+            )
+
+            enhanced_img = crop_fethear_ellipse(
+                main_frames[i],
+                30,
+                inpainting_mask_blur / 3 // 2,
+                inpainting_mask_blur / 3 // 2,
+            )
+            save2Collect(main_frames[i], out_config, f"main_frame_{i}")
+            save2Collect(enhanced_img, out_config, f"main_frame_enhanced_{i}")
+            corrected_frame.paste(enhanced_img, mask=enhanced_img)
+            main_frames[i] = corrected_frame
+        # else :TEST
+        # current_image.paste(prev_image, mask=prev_image)
+    return main_frames, processed
+
+
+def create_zoom(
+    common_prompt_pre,
+    prompts_array,
+    common_prompt_suf,
+    negative_prompt,
+    num_outpainting_steps,
+    guidance_scale,
+    num_inference_steps,
+    custom_init_image,
+    custom_exit_image,
+    video_frame_rate,
+    video_zoom_mode,
+    video_start_frame_dupe_amount,
+    video_last_frame_dupe_amount,
+    inpainting_mask_blur,
+    inpainting_fill_mode,
+    zoom_speed,
+    seed,
+    outputsizeW,
+    outputsizeH,
+    batchcount,
+    sampler,
+    upscale_do,
+    upscaler_name,
+    upscale_by,
+    inpainting_denoising_strength=1,
+    inpainting_full_res=0,
+    inpainting_padding=0,
+    progress=None,
+):
+    for i in range(batchcount):
+        print(f"Batch {i+1}/{batchcount}")
+        result = create_zoom_single(
+            common_prompt_pre,
+            prompts_array,
+            common_prompt_suf,
+            negative_prompt,
+            num_outpainting_steps,
+            guidance_scale,
+            num_inference_steps,
+            custom_init_image,
+            custom_exit_image,
+            video_frame_rate,
+            video_zoom_mode,
+            video_start_frame_dupe_amount,
+            video_last_frame_dupe_amount,
+            inpainting_mask_blur,
+            inpainting_fill_mode,
+            zoom_speed,
+            seed,
+            outputsizeW,
+            outputsizeH,
+            sampler,
+            upscale_do,
+            upscaler_name,
+            upscale_by,
+            inpainting_denoising_strength,
+            inpainting_full_res,
+            inpainting_padding,
+            progress,
+        )
+    return result
+
+
+def prepare_output_path():
+    isCollect = shared.opts.data.get("infzoom_collectAllResources", False)
+    output_path = shared.opts.data.get("infzoom_outpath", "outputs")
+
+    save_path = os.path.join(
+        output_path, shared.opts.data.get("infzoom_outSUBpath", "infinite-zooms")
+    )
+
+    if isCollect:
+        save_path = os.path.join(save_path, "iz_collect" + str(int(time.time())))
+
+    if not os.path.exists(save_path):
+        os.makedirs(save_path)
+
+    video_filename = os.path.join(
+        save_path, "infinite_zoom_" + str(int(time.time())) + ".mp4"
+    )
+
+    return {
+        "isCollect": isCollect,
+        "save_path": save_path,
+        "video_filename": video_filename,
+    }
+
+
+def save2Collect(img, out_config, name):
+    if out_config["isCollect"]:
+        img.save(f'{out_config["save_path"]}/{name}.png')
+
+
+def frame2Collect(all_frames, out_config):
+    save2Collect(all_frames[-1], out_config, f"frame_{len(all_frames)}")
+
+
+def frames2Collect(all_frames, out_config):
+    for i, f in enumerate(all_frames):
+        save2Collect(f, out_config, f"frame_{i}")
+
+
+def crop_inner_image(outpainted_img, width_offset, height_offset):
+    width, height = outpainted_img.size
+
+    center_x, center_y = int(width / 2), int(height / 2)
+
+    # Crop the image to the center
+    cropped_img = outpainted_img.crop(
+        (
+            center_x - width_offset,
+            center_y - height_offset,
+            center_x + width_offset,
+            center_y + height_offset,
+        )
+    )
+    prev_step_img = cropped_img.resize((width, height), resample=Image.LANCZOS)
+    # resized_img = resized_img.filter(ImageFilter.SHARPEN)
+
+    return prev_step_img
+
+
+def create_zoom_single(
+    common_prompt_pre,
+    prompts_array,
+    common_prompt_suf,
+    negative_prompt,
+    num_outpainting_steps,
+    guidance_scale,
+    num_inference_steps,
+    custom_init_image,
+    custom_exit_image,
+    video_frame_rate,
+    video_zoom_mode,
+    video_start_frame_dupe_amount,
+    video_last_frame_dupe_amount,
+    inpainting_mask_blur,
+    inpainting_fill_mode,
+    zoom_speed,
+    seed,
+    outputsizeW,
+    outputsizeH,
+    sampler,
+    upscale_do,
+    upscaler_name,
+    upscale_by,
+    inpainting_denoising_strength,
+    inpainting_full_res,
+    inpainting_padding,
+    progress,
+):
+    # try:
+    #     if gr.Progress() is not None:
+    #         progress = gr.Progress()
+    #         progress(0, desc="Preparing Initial Image")
+    # except Exception:
+    #     pass
+    fix_env_Path_ffprobe()
+    out_config = prepare_output_path()
+
+    prompts = {}
+
+    for x in prompts_array:
+        try:
+            key = int(x[0])
+            value = str(x[1])
+            prompts[key] = value
+        except ValueError:
+            pass
+
+    assert len(prompts_array) > 0, "prompts is empty"
+
+    width = closest_upper_divisible_by_eight(outputsizeW)
+    height = closest_upper_divisible_by_eight(outputsizeH)
+
+    current_image = Image.new(mode="RGBA", size=(width, height))
+    mask_image = np.array(current_image)[:, :, 3]
+    mask_image = Image.fromarray(255 - mask_image).convert("RGB")
+    current_image = current_image.convert("RGB")
+    current_seed = seed
+
+    if custom_init_image:
+        current_image = custom_init_image.resize(
+            (width, height), resample=Image.LANCZOS
+        )
+        save2Collect(current_image, out_config, f"init_custom.png")
+
+    else:
+        load_model_from_setting(
+            "infzoom_txt2img_model", progress, "Loading Model for txt2img: "
+        )
+
+        pr = prompts[min(k for k in prompts.keys() if k >= 0)]
+        processed, newseed = renderTxt2Img(
+            f"{common_prompt_pre}\n{pr}\n{common_prompt_suf}".strip(),
+            negative_prompt,
+            sampler,
+            num_inference_steps,
+            guidance_scale,
+            current_seed,
+            width,
+            height,
+        )
+        if len(processed.images) > 0:
+            current_image = processed.images[0]
+            save2Collect(current_image, out_config, f"init_txt2img.png")
+        current_seed = newseed
+
+    mask_width = math.trunc(width / 4)  # was initially 512px => 128px
+    mask_height = math.trunc(height / 4)  # was initially 512px => 128px
+
+    num_interpol_frames = round(video_frame_rate * zoom_speed)
+
+    all_frames = []
+
+    if upscale_do and progress:
+        progress(0, desc="upscaling inital image")
+
+    load_model_from_setting(
+        "infzoom_inpainting_model", progress, "Loading Model for inpainting/img2img: "
+    )
+    main_frames, processed = outpaint_steps(
+        width,
+        height,
+        common_prompt_pre,
+        common_prompt_suf,
+        prompts,
+        negative_prompt,
+        seed,
+        sampler,
+        num_inference_steps,
+        guidance_scale,
+        inpainting_denoising_strength,
+        inpainting_mask_blur,
+        inpainting_fill_mode,
+        inpainting_full_res,
+        inpainting_padding,
+        current_image,
+        num_outpainting_steps,
+        out_config,
+        mask_width,
+        mask_height,
+        custom_exit_image,
+    )
+    all_frames.append(
+        do_upscaleImg(main_frames[0], upscale_do, upscaler_name, upscale_by)
+        if upscale_do
+        else main_frames[0]
+    )
+    for i in range(len(main_frames) - 1):
+        # interpolation steps between 2 inpainted images (=sequential zoom and crop)
+        for j in range(num_interpol_frames - 1):
+            current_image = main_frames[i + 1]
+            interpol_image = current_image
+            save2Collect(interpol_image, out_config, f"interpol_img_{i}_{j}].png")
+
+            interpol_width = math.ceil(
+                (
+                    1
+                    - (1 - 2 * mask_width / width)
+                    ** (1 - (j + 1) / num_interpol_frames)
+                )
+                * width
+                / 2
+            )
+
+            interpol_height = math.ceil(
+                (
+                    1
+                    - (1 - 2 * mask_height / height)
+                    ** (1 - (j + 1) / num_interpol_frames)
+                )
+                * height
+                / 2
+            )
+
+            interpol_image = interpol_image.crop(
+                (
+                    interpol_width,
+                    interpol_height,
+                    width - interpol_width,
+                    height - interpol_height,
+                )
+            )
+
+            interpol_image = interpol_image.resize((width, height))
+            save2Collect(interpol_image, out_config, f"interpol_resize_{i}_{j}.png")
+
+            # paste the higher resolution previous image in the middle to avoid drop in quality caused by zooming
+            interpol_width2 = math.ceil(
+                (1 - (width - 2 * mask_width) / (width - 2 * interpol_width))
+                / 2
+                * width
+            )
+
+            interpol_height2 = math.ceil(
+                (1 - (height - 2 * mask_height) / (height - 2 * interpol_height))
+                / 2
+                * height
+            )
+
+            prev_image_fix_crop = shrink_and_paste_on_blank(
+                main_frames[i], interpol_width2, interpol_height2
+            )
+
+            interpol_image.paste(prev_image_fix_crop, mask=prev_image_fix_crop)
+            save2Collect(interpol_image, out_config, f"interpol_prevcrop_{i}_{j}.png")
+
+            if upscale_do and progress:
+                progress(((i + 1) / num_outpainting_steps), desc="upscaling interpol")
+
+            all_frames.append(
+                do_upscaleImg(interpol_image, upscale_do, upscaler_name, upscale_by)
+                if upscale_do
+                else interpol_image
+            )
+
+        if upscale_do and progress:
+            progress(((i + 1) / num_outpainting_steps), desc="upscaling current")
+
+        all_frames.append(
+            do_upscaleImg(current_image, upscale_do, upscaler_name, upscale_by)
+            if upscale_do
+            else current_image
+        )
+
+    frames2Collect(all_frames, out_config)
+
+    write_video(
+        out_config["video_filename"],
+        all_frames,
+        video_frame_rate,
+        video_zoom_mode,
+        int(video_start_frame_dupe_amount),
+        int(video_last_frame_dupe_amount),
+    )
+    print("Video saved in: " + os.path.join(script_path, out_config["video_filename"]))
+    return (
+        out_config["video_filename"],
+        main_frames,
+        processed.js(),
+        plaintext_to_html(processed.info),
+        plaintext_to_html(""),
+    )

infinite-zoom-automatic1111-webui/iz_helpers/sd_helpers.py

@@ -0,0 +1,81 @@
+from modules.processing import (
+    process_images,
+    StableDiffusionProcessingTxt2Img,
+    StableDiffusionProcessingImg2Img,
+)
+import modules.shared as shared
+
+
+def renderTxt2Img(
+    prompt, negative_prompt, sampler, steps, cfg_scale, seed, width, height
+):
+    processed = None
+    p = StableDiffusionProcessingTxt2Img(
+        sd_model=shared.sd_model,
+        outpath_samples=shared.opts.outdir_txt2img_samples,
+        outpath_grids=shared.opts.outdir_txt2img_grids,
+        prompt=prompt,
+        negative_prompt=negative_prompt,
+        seed=seed,
+        sampler_name=sampler,
+        n_iter=1,
+        steps=steps,
+        cfg_scale=cfg_scale,
+        width=width,
+        height=height,
+    )
+    processed = process_images(p)
+    newseed = p.seed
+    return processed, newseed
+
+
+def renderImg2Img(
+    prompt,
+    negative_prompt,
+    sampler,
+    steps,
+    cfg_scale,
+    seed,
+    width,
+    height,
+    init_image,
+    mask_image,
+    inpainting_denoising_strength,
+    inpainting_mask_blur,
+    inpainting_fill_mode,
+    inpainting_full_res,
+    inpainting_padding,
+):
+    processed = None
+
+    p = StableDiffusionProcessingImg2Img(
+        sd_model=shared.sd_model,
+        outpath_samples=shared.opts.outdir_img2img_samples,
+        outpath_grids=shared.opts.outdir_img2img_grids,
+        prompt=prompt,
+        negative_prompt=negative_prompt,
+        seed=seed,
+        sampler_name=sampler,
+        n_iter=1,
+        steps=steps,
+        cfg_scale=cfg_scale,
+        width=width,
+        height=height,
+        init_images=[init_image],
+        denoising_strength=inpainting_denoising_strength,
+        mask_blur=inpainting_mask_blur,
+        inpainting_fill=inpainting_fill_mode,
+        inpaint_full_res=inpainting_full_res,
+        inpaint_full_res_padding=inpainting_padding,
+        mask=mask_image,
+    )
+    # p.latent_mask = Image.new("RGB", (p.width, p.height), "white")
+    
+    processed = process_images(p)
+    # For those that use Image grids this will make sure that ffmpeg does not crash out
+    if (len(processed.images) > 1) and (processed.images[0].size[0] != processed.images[-1].size[0]):
+        processed.images.pop(0)
+        print("\nGrid image detected applying patch")
+    
+    newseed = p.seed
+    return processed, newseed

infinite-zoom-automatic1111-webui/iz_helpers/settings.py

@@ -0,0 +1,108 @@
+import gradio as gr
+import modules.shared as shared
+from .static_variables import default_prompt
+
+
+def on_ui_settings():
+    section = ("infinite-zoom", "Infinite Zoom")
+
+    shared.opts.add_option(
+        "infzoom_outpath",
+        shared.OptionInfo(
+            "outputs",
+            "Path where to store your infinite video. Default is Outputs",
+            gr.Textbox,
+            {"interactive": True},
+            section=section,    
+        ),
+    )
+
+    shared.opts.add_option(
+        "infzoom_outSUBpath",
+        shared.OptionInfo(
+            "infinite-zooms",
+            "Which subfolder name to be created in the outpath. Default is 'infinite-zooms'",
+            gr.Textbox,
+            {"interactive": True},
+            section=section,
+        ),
+    )
+
+    shared.opts.add_option(
+        "infzoom_outsizeW",
+        shared.OptionInfo(
+            512,
+            "Default width of your video",
+            gr.Slider,
+            {"minimum": 16, "maximum": 2048, "step": 16},
+            section=section,
+        ),
+    )
+
+    shared.opts.add_option(
+        "infzoom_outsizeH",
+        shared.OptionInfo(
+            512,
+            "Default height your video",
+            gr.Slider,
+            {"minimum": 16, "maximum": 2048, "step": 16},
+            section=section,
+        ),
+    )
+
+    shared.opts.add_option(
+        "infzoom_ffprobepath",
+        shared.OptionInfo(
+            "",
+            "Writing videos has  dependency to an existing FFPROBE executable on your machine. D/L here (https://github.com/BtbN/FFmpeg-Builds/releases) your OS variant and point to your installation path",
+            gr.Textbox,
+            {"interactive": True},
+            section=section,
+        ),
+    )
+
+    shared.opts.add_option(
+        "infzoom_txt2img_model",
+        shared.OptionInfo(
+            None,
+            "Name of your desired model to render keyframes (txt2img)",
+            gr.Dropdown,
+            lambda: {"choices": [x for x in list(shared.list_checkpoint_tiles()) if "inpainting" not in x]},
+            section=section,
+        ),
+    )
+
+    shared.opts.add_option(
+        "infzoom_inpainting_model",
+        shared.OptionInfo(
+            None,
+            "Name of your desired inpaint model (img2img-inpaint). Default is vanilla sd-v1-5-inpainting.ckpt ",
+            gr.Dropdown,
+            lambda: {"choices": [x for x in list(shared.list_checkpoint_tiles()) if "inpainting" in x]},
+            section=section,
+        ),
+    )
+
+    shared.opts.add_option(
+        "infzoom_defPrompt",
+        shared.OptionInfo(
+            default_prompt,
+            "Default prompt-setup to start with'",
+            gr.Code,
+            {"interactive": True, "language": "json"},
+            section=section,
+        ),
+    )
+    
+    
+    shared.opts.add_option(
+        "infzoom_collectAllResources",
+        shared.OptionInfo(
+            False,
+            "!!! Store all images (txt2img, init_image,exit_image, inpainting, interpolation) into one folder in your OUTPUT Path. Very slow, a lot of data. Dont do this on long runs !!!",
+            gr.Checkbox,
+            {"interactive": True},
+            section=section,
+        ),
+    )
+     

infinite-zoom-automatic1111-webui/iz_helpers/static_variables.py

@@ -0,0 +1,51 @@
+import os
+from modules import scripts
+import modules.sd_samplers
+
+default_sampling_steps = 35
+default_sampler = "DDIM"
+default_cfg_scale = 8
+default_mask_blur = 48
+default_total_outpaints = 5
+promptTableHeaders = ["Start at second [0,1,...]", "Prompt"]
+
+default_prompt = """
+{
+	"prePrompt": "Huge spectacular Waterfall in ",
+	"prompts": {
+		"data": [
+			[0, "a dense tropical forest"],
+			[2, "a Lush jungle"],
+			[3, "a Thick rainforest"],
+			[5, "a Verdant canopy"]
+		]
+	},
+	"postPrompt": "epic perspective,(vegetation overgrowth:1.3)(intricate, ornamentation:1.1),(baroque:1.1), fantasy, (realistic:1) digital painting , (magical,mystical:1.2) , (wide angle shot:1.4), (landscape composed:1.2)(medieval:1.1),(tropical forest:1.4),(river:1.3) volumetric lighting ,epic, style by Alex Horley Wenjun Lin greg rutkowski Ruan Jia (Wayne Barlowe:1.2)",
+	"negPrompt": "frames, border, edges, borderline, text, character, duplicate, error, out of frame, watermark, low quality, ugly, deformed, blur, bad-artist"
+}
+"""
+
+empty_prompt = '{"prompts":{"data":[],"negPrompt":"", prePrompt:"", postPrompt:""}'
+
+invalid_prompt = {
+    "prompts": {
+        "data": [[0, "Your prompt-json is invalid, please check Settings"]],
+    },
+    "negPrompt": "Invalid prompt-json",
+    "prePrompt": "Invalid prompt",
+    "postPrompt": "Invalid prompt",
+}
+
+available_samplers = [
+    s.name for s in modules.sd_samplers.samplers if "UniPc" not in s.name
+]
+
+current_script_dir = scripts.basedir().split(os.sep)[
+    -2:
+]  # contains install and our extension foldername
+jsonprompt_schemafile = (
+    current_script_dir[0]
+    + "/"
+    + current_script_dir[1]
+    + "/iz_helpers/promptschema.json"
+)

infinite-zoom-automatic1111-webui/iz_helpers/ui.py

@@ -0,0 +1,307 @@
+import gradio as gr
+from .run import create_zoom
+import modules.shared as shared
+from webui import wrap_gradio_gpu_call
+from modules.ui import create_output_panel
+
+from .static_variables import (
+    default_prompt,
+    available_samplers,
+    default_total_outpaints,
+    default_sampling_steps,
+    default_cfg_scale,
+    default_mask_blur,
+    default_sampler,
+)
+from .helpers import putPrompts, clearPrompts
+from .prompt_util import readJsonPrompt
+from .static_variables import promptTableHeaders
+
+
+def on_ui_tabs():
+    with gr.Blocks(analytics_enabled=False) as infinite_zoom_interface:
+        gr.HTML(
+            """
+            <p style="text-align: center;">
+                <a target="_blank" href="https://github.com/v8hid/infinite-zoom-automatic1111-webui"><img src="https://img.shields.io/static/v1?label=github&message=repository&color=blue&style=flat&logo=github&logoColor=white" style="display: inline;" alt="GitHub Repo"/></a>
+                <a href="https://discord.gg/v2nHqSrWdW"><img src="https://img.shields.io/discord/1095469311830806630?color=blue&label=discord&logo=discord&logoColor=white" style="display: inline;" alt="Discord server"></a>
+            </p>
+
+            """
+        )
+        with gr.Row():
+            generate_btn = gr.Button(value="Generate video", variant="primary")
+            interrupt = gr.Button(value="Interrupt", elem_id="interrupt_training")
+        with gr.Row():
+            with gr.Column(scale=1, variant="panel"):
+                with gr.Tab("Main"):
+                    with gr.Row():
+                        batchcount_slider = gr.Slider(
+                            minimum=1,
+                            maximum=25,
+                            value=shared.opts.data.get("infzoom_batchcount", 1),
+                            step=1,
+                            label="Batch Count",
+                        )
+                        main_outpaint_steps = gr.Number(
+                            label="Total video length [s]",
+                            value=default_total_outpaints,
+                            precision=0,
+                            interactive=True,
+                        )
+
+                    # safe reading json prompt
+                    pr = shared.opts.data.get("infzoom_defPrompt", default_prompt)
+                    jpr = readJsonPrompt(pr, True)
+
+                    main_common_prompt_pre = gr.Textbox(
+                        value=jpr["prePrompt"], label="Common Prompt Prefix"
+                    )
+
+                    main_prompts = gr.Dataframe(
+                        type="array",
+                        headers=promptTableHeaders,
+                        datatype=["number", "str"],
+                        row_count=1,
+                        col_count=(2, "fixed"),
+                        value=jpr["prompts"],
+                        wrap=True,
+                    )
+
+                    main_common_prompt_suf = gr.Textbox(
+                        value=jpr["postPrompt"], label="Common Prompt Suffix"
+                    )
+
+                    main_negative_prompt = gr.Textbox(
+                        value=jpr["negPrompt"], label="Negative Prompt"
+                    )
+
+                    # these button will be moved using JS under the dataframe view as small ones
+                    exportPrompts_button = gr.Button(
+                        value="Export prompts",
+                        variant="secondary",
+                        elem_classes="sm infzoom_tab_butt",
+                        elem_id="infzoom_exP_butt",
+                    )
+                    importPrompts_button = gr.UploadButton(
+                        label="Import prompts",
+                        variant="secondary",
+                        elem_classes="sm infzoom_tab_butt",
+                        elem_id="infzoom_imP_butt",
+                    )
+                    exportPrompts_button.click(
+                        None,
+                        _js="exportPrompts",
+                        inputs=[
+                            main_common_prompt_pre,
+                            main_prompts,
+                            main_common_prompt_suf,
+                            main_negative_prompt,
+                        ],
+                        outputs=None,
+                    )
+                    importPrompts_button.upload(
+                        fn=putPrompts,
+                        outputs=[
+                            main_common_prompt_pre,
+                            main_prompts,
+                            main_common_prompt_suf,
+                            main_negative_prompt,
+                        ],
+                        inputs=[importPrompts_button],
+                    )
+
+                    clearPrompts_button = gr.Button(
+                        value="Clear prompts",
+                        variant="secondary",
+                        elem_classes="sm infzoom_tab_butt",
+                        elem_id="infzoom_clP_butt",
+                    )
+                    clearPrompts_button.click(
+                        fn=clearPrompts,
+                        inputs=[],
+                        outputs=[
+                            main_prompts,
+                            main_negative_prompt,
+                            main_common_prompt_pre,
+                            main_common_prompt_suf,
+                        ],
+                    )
+
+                    with gr.Accordion("Render settings"):
+                        with gr.Row():
+                            seed = gr.Number(
+                                label="Seed", value=-1, precision=0, interactive=True
+                            )
+                            main_sampler = gr.Dropdown(
+                                label="Sampler",
+                                choices=available_samplers,
+                                value=default_sampler,
+                                type="value",
+                            )
+                        with gr.Row():
+                            main_width = gr.Slider(
+                                minimum=16,
+                                maximum=2048,
+                                value=shared.opts.data.get("infzoom_outsizeW", 512),
+                                step=16,
+                                label="Output Width",
+                            )
+                            main_height = gr.Slider(
+                                minimum=16,
+                                maximum=2048,
+                                value=shared.opts.data.get("infzoom_outsizeH", 512),
+                                step=16,
+                                label="Output Height",
+                            )
+                        with gr.Row():
+                            main_guidance_scale = gr.Slider(
+                                minimum=0.1,
+                                maximum=15,
+                                step=0.1,
+                                value=default_cfg_scale,
+                                label="Guidance Scale",
+                            )
+                            sampling_step = gr.Slider(
+                                minimum=1,
+                                maximum=150,
+                                step=1,
+                                value=default_sampling_steps,
+                                label="Sampling Steps for each outpaint",
+                            )
+                        with gr.Row():
+                            init_image = gr.Image(
+                                type="pil", label="Custom initial image"
+                            )
+                            exit_image = gr.Image(
+                                type="pil", label="Custom exit image", visible=False
+                            )
+                with gr.Tab("Video"):
+                    video_frame_rate = gr.Slider(
+                        label="Frames per second",
+                        value=30,
+                        minimum=1,
+                        maximum=60,
+                    )
+                    video_zoom_mode = gr.Radio(
+                        label="Zoom mode",
+                        choices=["Zoom-out", "Zoom-in"],
+                        value="Zoom-out",
+                        type="index",
+                    )
+                    video_start_frame_dupe_amount = gr.Slider(
+                        label="number of start frame dupe",
+                        info="Frames to freeze at the start of the video",
+                        value=0,
+                        minimum=1,
+                        maximum=60,
+                    )
+                    video_last_frame_dupe_amount = gr.Slider(
+                        label="number of last frame dupe",
+                        info="Frames to freeze at the end of the video",
+                        value=0,
+                        minimum=1,
+                        maximum=60,
+                    )
+                    video_zoom_speed = gr.Slider(
+                        label="Zoom Speed",
+                        value=1.0,
+                        minimum=0.1,
+                        maximum=20.0,
+                        step=0.1,
+                        info="Zoom speed in seconds (higher values create slower zoom)",
+                    )
+
+                with gr.Tab("Outpaint"):
+                    inpainting_mask_blur = gr.Slider(
+                        label="Mask Blur",
+                        minimum=0,
+                        maximum=64,
+                        value=default_mask_blur,
+                    )
+                    inpainting_fill_mode = gr.Radio(
+                        label="Masked content",
+                        choices=["fill", "original", "latent noise", "latent nothing"],
+                        value="latent noise",
+                        type="index",
+                    )
+
+                with gr.Tab("Post proccess"):
+                    upscale_do = gr.Checkbox(False, label="Enable Upscale")
+                    upscaler_name = gr.Dropdown(
+                        label="Upscaler",
+                        elem_id="infZ_upscaler",
+                        choices=[x.name for x in shared.sd_upscalers],
+                        value=shared.sd_upscalers[0].name,
+                    )
+                    upscale_by = gr.Slider(
+                        label="Upscale by factor",
+                        minimum=1,
+                        maximum=8,
+                        step=0.5,
+                        value=2,
+                    )
+                    with gr.Accordion("Help", open=False):
+                        gr.Markdown(
+                            """# Performance critical
+Depending on amount of frames and which upscaler you choose it might took a long time to render.  
+Our best experience and trade-off is the R-ERSGAn4x upscaler.
+"""
+                        )
+
+            with gr.Column(scale=1, variant="compact"):
+                output_video = gr.Video(label="Output").style(width=512, height=512)
+                (
+                    out_image,
+                    generation_info,
+                    html_info,
+                    html_log,
+                ) = create_output_panel(
+                    "infinite-zoom", shared.opts.outdir_img2img_samples
+                )
+
+        generate_btn.click(
+            fn=wrap_gradio_gpu_call(create_zoom, extra_outputs=[None, "", ""]),
+            inputs=[
+                main_common_prompt_pre,
+                main_prompts,
+                main_common_prompt_suf,
+                main_negative_prompt,
+                main_outpaint_steps,
+                main_guidance_scale,
+                sampling_step,
+                init_image,
+                exit_image,
+                video_frame_rate,
+                video_zoom_mode,
+                video_start_frame_dupe_amount,
+                video_last_frame_dupe_amount,
+                inpainting_mask_blur,
+                inpainting_fill_mode,
+                video_zoom_speed,
+                seed,
+                main_width,
+                main_height,
+                batchcount_slider,
+                main_sampler,
+                upscale_do,
+                upscaler_name,
+                upscale_by,
+            ],
+            outputs=[output_video, out_image, generation_info, html_info, html_log],
+        )
+
+        main_prompts.change(
+            fn=checkPrompts, inputs=[main_prompts], outputs=[generate_btn]
+        )
+
+        interrupt.click(fn=lambda: shared.state.interrupt(), inputs=[], outputs=[])
+    infinite_zoom_interface.queue()
+    return [(infinite_zoom_interface, "Infinite Zoom", "iz_interface")]
+
+
+def checkPrompts(p):
+    return gr.Button.update(
+        interactive=any(0 in sublist for sublist in p)
+        or any("0" in sublist for sublist in p)
+    )

infinite-zoom-automatic1111-webui/iz_helpers/video.py

@@ -0,0 +1,43 @@
+import numpy as np
+import imageio
+from PIL import Image
+
+def write_video(file_path, frames, fps, reversed=True, start_frame_dupe_amount=15, last_frame_dupe_amount=30):
+    """
+    Writes frames to an mp4 video file
+    :param file_path: Path to output video, must end with .mp4
+    :param frames: List of PIL.Image objects
+    :param fps: Desired frame rate
+    :param reversed: if order of images to be reversed (default = True)
+    """
+    if reversed == True:
+        frames = frames[::-1]
+
+    # Drop missformed frames
+    frames = [frame for frame in frames if frame.size == frames[0].size]
+
+    # Create an imageio video writer, avoid block size of 512.
+    writer = imageio.get_writer(file_path, fps=fps, macro_block_size=None)
+
+    # Duplicate the start and end frames
+    start_frames = [frames[0]] * start_frame_dupe_amount
+    end_frames = [frames[-1]] * last_frame_dupe_amount
+
+    # Write the duplicated frames to the video writer
+    for frame in start_frames:
+        # Convert PIL image to numpy array
+        np_frame = np.array(frame)
+        writer.append_data(np_frame)
+
+    # Write the frames to the video writer
+    for frame in frames:
+        np_frame = np.array(frame)
+        writer.append_data(np_frame)
+
+    # Write the duplicated frames to the video writer
+    for frame in  end_frames:
+        np_frame = np.array(frame)
+        writer.append_data(np_frame)
+
+    # Close the video writer
+    writer.close()

infinite-zoom-automatic1111-webui/javascript/infinite-zoom-hints.js

@@ -0,0 +1,49 @@
+// mouseover tooltips for various UI elements
+
+infzoom_titles = {
+    "Batch Count":"How many separate videos to create",
+    "Total video length [s]":"For each seconds frame (FPS) will be generated. Define prompts at which time they should start wihtin this duration.",
+    "Common Prompt Prefix":"Prompt inserted before each step",
+    "Common Prompt Suffix":"Prompt inserted after each step",
+    "Negative Prompt":"What your model shall avoid",
+    "Export prompts": "Downloads a JSON file to save all prompts",
+    "Import prompts": "Restore Prompts table from a specific JSON file",
+    "Clear prompts": "Start over, remove all entries from prompt table, prefix, suffix, negative",
+    "Custom initial image":"An image at the end resp. begin of your movie, depending or ZoomIn or Out",
+    "Custom exit image":"An image at the end resp. begin of your movie, depending or ZoomIn or Out",
+    "Zoom Speed":"Varies additional frames per second",
+    "Start at second [0,1,...]": "At which time the prompt has to be occure. We need at least one prompt starting at time 0",
+	"Generate video": "Start rendering. If it´s disabled the prompt table is invalid, check we have a start prompt at time 0"
+}
+
+
+onUiUpdate(function(){
+	gradioApp().querySelectorAll('span, button, select, p').forEach(function(span){
+		tooltip = infzoom_titles[span.textContent];
+
+		if(!tooltip){
+		    tooltip = infzoom_titles[span.value];
+		}
+
+		if(!tooltip){
+			for (const c of span.classList) {
+				if (c in infzoom_titles) {
+					tooltip = infzoom_titles[c];
+					break;
+				}
+			}
+		}
+
+		if(tooltip){
+			span.title = tooltip;
+		}
+	})
+
+	gradioApp().querySelectorAll('select').forEach(function(select){
+	    if (select.onchange != null) return;
+
+	    select.onchange = function(){
+            select.title = infzoom_titles[select.value] || "";
+	    }
+	})
+})

infinite-zoom-automatic1111-webui/javascript/infinite-zoom.js

@@ -0,0 +1,54 @@
+// Function to download data to a file
+function exportPrompts(cppre,p, cpsuf,np, filename = "infinite-zoom-prompts.json") {
+
+    let J = { prompts: p, negPrompt: np, prePrompt: cppre, postPrompt: cpsuf }
+
+    var file = new Blob([JSON.stringify(J,null,2)], { type: "text/csv" });
+    if (window.navigator.msSaveOrOpenBlob) // IE10+
+        window.navigator.msSaveOrOpenBlob(file, filename);
+    else { // Others
+        var a = document.createElement("a"),
+            url = URL.createObjectURL(file);
+        a.href = url;
+        a.download = filename;
+        document.body.appendChild(a);
+        a.click();
+        setTimeout(function () {
+            document.body.removeChild(a);
+            window.URL.revokeObjectURL(url);
+        }, 0);
+    }
+}
+
+document.addEventListener("DOMContentLoaded", () => {
+    const onload = () => {
+
+        if (typeof gradioApp === "function") {
+            /* move big buttons directly under the tabl of prompts as SMall ones */
+            const wrap = gradioApp().querySelector("#tab_iz_interface .gradio-dataframe .controls-wrap")
+
+            if (wrap) {
+                let butts = gradioApp().querySelectorAll("#tab_iz_interface .infzoom_tab_butt")
+                butts.forEach(b => {
+                    wrap.appendChild(b)
+                    b.classList.replace("lg", "sm") // smallest
+                });
+            }
+            else {
+                setTimeout(onload, 2000);
+            }
+        }
+        else {
+            setTimeout(onload, 2000);
+        }
+    };
+    onload();
+});
+
+
+
+
+
+
+
+

infinite-zoom-automatic1111-webui/scripts/infinite-zoom.py

@@ -0,0 +1,5 @@
+from modules import script_callbacks
+from iz_helpers.ui import on_ui_tabs
+from iz_helpers.settings import on_ui_settings
+script_callbacks.on_ui_tabs(on_ui_tabs)
+script_callbacks.on_ui_settings(on_ui_settings)

infinite-zoom-automatic1111-webui/style.css

@@ -0,0 +1,10 @@
+#tab_iz_interface .gradio-dataframe .controls-wrap {
+    flex-direction: row-reverse;
+    justify-content: space-between;
+}
+
+/* first column min width */
+#tab_iz_interface th:first-child  {
+    flex: 0 0 0%;
+    width: 0;
+}

sd-webui-cutoff/.gitignore

@@ -0,0 +1 @@
+__pycache__

sd-webui-cutoff/LICENSE

@@ -0,0 +1 @@
+MIT License

sd-webui-cutoff/README.md

@@ -0,0 +1,146 @@
+# Cutoff - Cutting Off Prompt Effect
+
+![cover](./images/cover.jpg)
+
+<details>
+<summary>Update Info</summary>
+
+Upper is newer.
+
+<dl>
+<dt>20e87ce264338b824296b7559679ed1bb0bdacd7</dt>
+<dd>Skip empty targets.</dd>
+<dt>03bfe60162ba418e18dbaf8f1b9711fd62195ef3</dt>
+<dd>Add <code>Disable for Negative prompt</code> option. Default is <code>True</code>.</dd>
+<dt>f0990088fed0f5013a659cacedb194313a398860</dt>
+<dd>Accept an empty prompt.</dd>
+</dl>
+</details>
+
+## What is this?
+
+This is an extension for [stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui) which limits the tokens' influence scope.
+
+## Usage
+
+1. Select `Enabled` checkbox.
+2. Input words which you want to limit scope in `Target tokens`.
+3. Generate images.
+
+## Note
+
+If the generated image was corrupted or something like that, try to change the `Weight` value or change the interpolation method to `SLerp`. Interpolation method can be found in `Details`.
+
+### `Details` section
+
+<dl>
+<dt>Disable for Negative prompt.</dt>
+<dd>If enabled, <b>Cutoff</b> will not work for the negative prompt. Default is <code>true</code>.</dd>
+<dt>Cutoff strongly.</dt>
+<dd>See <a href="#how-it-works">description below</a>. Default is <code>false</code>.</dd>
+<dt>Interpolation method</dt>
+<dd>How "padded" and "original" vectors will be interpolated. Default is <code>Lerp</code>.</dd>
+<dt>Padding token</dt>
+<dd>What token will be padded instead of <code>Target tokens</code>. Default is <code>_</code> (underbar).</dd>
+</dl>
+
+## Examples
+
+```
+7th_anime_v3_A-fp16 / kl-f8-anime2 / DPM++ 2M Karras / 15 steps / 512x768
+Prompt: a cute girl, white shirt with green tie, red shoes, blue hair, yellow eyes, pink skirt
+Negative Prompt: (low quality, worst quality:1.4), nsfw
+Target tokens: white, green, red, blue, yellow, pink
+```
+
+Sample 1.
+
+![sample 1](./images/sample-1.png)
+
+Sample 2. (use `SLerp` for interpolation)
+
+![sample 2](./images/sample-2.png)
+
+Sample 3.
+
+![sample 3](./images/sample-3.png)
+
+## How it works
+
+- [Japanese](#japanese)
+- [English](#english)
+
+or see [#5](https://github.com/hnmr293/sd-webui-cutoff/issues/5).
+
+![idea](./images/idea.png)
+
+### Japanese
+
+プロンプトをCLIPに通して得られる (77, 768) 次元の埋め込み表現（？正式な用語は分かりません）について、
+ごく単純には、77個の行ベクトルはプロンプト中の75個のトークン（＋開始トークン＋終了トークン）に対応していると考えられる。
+
+※上図は作図上、この説明とは行と列を入れ替えて描いている。
+
+このベクトルには単語単体の意味だけではなく、文章全体の、例えば係り結びなどの情報を集約したものが入っているはずである。
+
+ここで `a cute girl, pink hair, red shoes` というプロンプトを考える。
+普通、こういったプロンプトの意図は
+
+1. `pink` は `hair` だけに係っており `shoes` には係っていない。
+2. 同様に `red` も `hair` には係っていない。
+3. `a cute girl` は全体に係っていて欲しい。`hair` や `shoes` は女の子に合うものが出て欲しい。
+
+……というもののはずである。
+
+しかしながら、[EvViz2](https://github.com/hnmr293/sd-webui-evviz2) などでトークン間の関係を見ると、そううまくはいっていないことが多い。
+つまり、`shoes` の位置のベクトルに `pink` の影響が出てしまっていたりする。
+
+一方で上述の通り `a cute girl` の影響は乗っていて欲しいわけで、どうにかして、特定のトークンの影響を取り除けるようにしたい。
+
+この拡張では、指定されたトークンを *padding token* に書き換えることでそれを実現している。
+
+たとえば `red shoes` の部分に対応して `a cute girl, _ hair, red shoes` というプロンプトを生成する。`red` と `shoes` に対応する位置のベクトルをここから生成したもので上書きしてやることで、`pink` の影響を除外している。
+
+これを `pink` の側から見ると、自分の影響が `pink hair` の範囲内に制限されているように見える。What is this? の "limits the tokens' influence scope" はそういう意味。
+
+ところで `a cute girl` の方は、`pink hair, red shoes` の影響を受けていてもいいし受けなくてもいいような気がする。
+そこでこの拡張では、こういうどちらでもいいプロンプトに対して
+
+1. `a cute girl, pink hair, red shoes`
+2. `a cute girl, _ hair, _ shoes`
+
+のどちらを適用するか選べるようにしている。`Details` の `Cutoff strongly` がそれで、オフのとき1.を、オンのとき2.を、それぞれ選ぶようになっている。
+元絵に近いのが出るのはオフのとき。デフォルトもこちらにしてある。
+
+### English
+
+NB. The following text is a translation of the Japanese text above by [DeepL](https://www.deepl.com/translator).
+
+For the (77, 768) dimensional embedded representation (I don't know the formal terminology), one could simply assume that the 77 row vectors correspond to the 75 tokens (+ start token and end token) in the prompt.
+
+Note: The above figure is drawn with the rows and columns interchanged from this explanation.
+
+This vector should contain not only the meanings of individual words, but also the aggregate information of the whole sentence, for example, the connection between words.
+
+Consider the prompt `a cute girl, pink hair, red shoes`. Normally, the intent of such a prompt would be
+
+- `pink` is only for `hair`, not `shoes`.
+- Similarly, `red` does not refer to `hair`.
+- We want `a cute girl` to be about the whole thing, and we want the `hair` and `shoes` to match the girl.
+
+However, when we look at the relationship between tokens in [EvViz2](https://github.com/hnmr293/sd-webui-evviz2) and other tools, we see that it is not always that way. In other words, the position vector of the `shoes` may be affected by `pink`.
+
+On the other hand, as mentioned above, we want the influence of `a cute girl` to be present, so we want to be able to somehow remove the influence of a specific token.
+
+This extension achieves this by rewriting the specified tokens as a *padding token*.
+
+For example, for the `red shoes` part, we generate the prompt `a cute girl, _ hair, red shoes`, and by overwriting the position vectors corresponding to `red` and `shoes` with those generated from this prompt, we remove the influence of `pink`.
+
+From `pink`'s point of view, it appears that its influence is limited to the `pink hair`'s scope.
+
+By the way, `a cute girl` may or may not be influenced by `pink hair` and `red shoes`. So, in this extension, for such a prompt that can be either
+
+1. `a cute girl, pink hair, red shoes`
+2. `a cute girl, _ hair, _ shoes`
+
+The `Cutoff strongly` in the `Details` section allows you to select 1 when it is off and 2 when it is on. The one that comes out closer to the original image is "off". The default is also set this way.

sd-webui-cutoff/scripts/cutoff.py

@@ -0,0 +1,257 @@
+from collections import defaultdict
+from typing import Union, List, Tuple
+
+import numpy as np
+import torch
+from torch import Tensor, nn
+import gradio as gr
+
+from modules.processing import StableDiffusionProcessing
+from modules import scripts
+
+from scripts.cutofflib.sdhook import SDHook
+from scripts.cutofflib.embedding import CLIP, generate_prompts, token_to_block
+from scripts.cutofflib.utils import log, set_debug
+from scripts.cutofflib.xyz import init_xyz
+
+NAME = 'Cutoff'
+PAD = '_</w>'
+
+def check_neg(s: str, negative_prompt: str, all_negative_prompts: Union[List[str],None]):
+    if s == negative_prompt:
+        return True
+    
+    if all_negative_prompts is not None:
+        return s in all_negative_prompts
+    
+    return False
+
+def slerp(t, v0, v1, DOT_THRESHOLD=0.9995):
+    # cf. https://memo.sugyan.com/entry/2022/09/09/230645
+
+    inputs_are_torch = False
+    input_device = v0.device
+    if not isinstance(v0, np.ndarray):
+        inputs_are_torch = True
+        v0 = v0.cpu().numpy()
+        v1 = v1.cpu().numpy()
+
+    dot = np.sum(v0 * v1 / (np.linalg.norm(v0) * np.linalg.norm(v1)))
+    if np.abs(dot) > DOT_THRESHOLD:
+        v2 = (1 - t) * v0 + t * v1
+    else:
+        theta_0 = np.arccos(dot)
+        sin_theta_0 = np.sin(theta_0)
+        theta_t = theta_0 * t
+        sin_theta_t = np.sin(theta_t)
+        s0 = np.sin(theta_0 - theta_t) / sin_theta_0
+        s1 = sin_theta_t / sin_theta_0
+        v2 = s0 * v0 + s1 * v1
+
+    if inputs_are_torch:
+        v2 = torch.from_numpy(v2).to(input_device)
+
+    return v2
+
+
+class Hook(SDHook):
+    
+    def __init__(
+        self,
+        enabled: bool,
+        targets: List[str],
+        padding: Union[str,int],
+        weight: float,
+        disable_neg: bool,
+        strong: bool,
+        interpolate: str,
+    ):
+        super().__init__(enabled)
+        self.targets = targets
+        self.padding = padding
+        self.weight = float(weight)
+        self.disable_neg = disable_neg
+        self.strong = strong
+        self.intp = interpolate
+    
+    def interpolate(self, t1: Tensor, t2: Tensor, w):
+        if self.intp == 'lerp':
+            return torch.lerp(t1, t2, w)
+        else:
+            return slerp(w, t1, t2)
+    
+    def hook_clip(self, p: StableDiffusionProcessing, clip: nn.Module):
+        
+        skip = False
+        
+        def hook(mod: nn.Module, inputs: Tuple[List[str]], output: Tensor):
+            nonlocal skip
+            
+            if skip:
+                # called from <A> below
+                return
+            
+            assert isinstance(mod, CLIP)
+            
+            prompts, *rest = inputs
+            assert len(prompts) == output.shape[0]
+            
+            # Check wether we are processing Negative prompt or not.
+            # I firmly believe there is no one who uses a negative prompt 
+            # exactly identical to a prompt.
+            if self.disable_neg:
+                if all(check_neg(x, p.negative_prompt, p.all_negative_prompts) for x in prompts):
+                    # Now we are processing Negative prompt and skip it.
+                    return
+            
+            output = output.clone()
+            for pidx, prompt in enumerate(prompts):
+                tt = token_to_block(mod, prompt)
+                
+                cutoff = generate_prompts(mod, prompt, self.targets, self.padding)
+                switch_base = np.full_like(cutoff.sw, self.strong)
+                switch = np.full_like(cutoff.sw, True)
+                active = cutoff.active_blocks()
+                
+                prompt_to_tokens = defaultdict(lambda: [])
+                for tidx, (token, block_index) in enumerate(tt):
+                    if block_index in active:
+                        sw = switch.copy()
+                        sw[block_index] = False
+                        prompt = cutoff.text(sw)
+                    else:
+                        prompt = cutoff.text(switch_base)
+                    prompt_to_tokens[prompt].append((tidx, token))
+                
+                #log(prompt_to_tokens)
+                
+                ks = list(prompt_to_tokens.keys())
+                if len(ks) == 0:
+                    # without any (negative) prompts
+                    ks.append('')
+                
+                try:
+                    # <A>
+                    skip = True
+                    vs = mod(ks)
+                finally:
+                    skip = False
+                
+                tensor = output[pidx, :, :] # e.g. (77, 768)
+                for k, t in zip(ks, vs):
+                    assert tensor.shape == t.shape
+                    for tidx, token in prompt_to_tokens[k]:
+                        log(f'{tidx:03} {token.token:<16} {k}')
+                        tensor[tidx, :] = self.interpolate(tensor[tidx,:], t[tidx,:], self.weight)
+                
+            return output
+        
+        self.hook_layer(clip, hook)
+    
+
+class Script(scripts.Script):
+    
+    def __init__(self):
+        super().__init__()
+        self.last_hooker: Union[SDHook,None] = None
+
+    def title(self):
+        return NAME
+    
+    def show(self, is_img2img):
+        return scripts.AlwaysVisible
+    
+    def ui(self, is_img2img):
+        with gr.Accordion(NAME, open=False):
+            enabled = gr.Checkbox(label='Enabled', value=False)
+            targets = gr.Textbox(label='Target tokens (comma separated)', placeholder='red, blue')
+            weight = gr.Slider(minimum=-1.0, maximum=2.0, step=0.01, value=0.5, label='Weight')
+            with gr.Accordion('Details', open=False):
+                disable_neg = gr.Checkbox(value=True, label='Disable for Negative prompt.')
+                strong = gr.Checkbox(value=False, label='Cutoff strongly.')
+                padding = gr.Textbox(label='Padding token (ID or single token)')
+                lerp = gr.Radio(choices=['Lerp', 'SLerp'], value='Lerp', label='Interpolation method')
+            
+            debug = gr.Checkbox(value=False, label='Debug log')
+            debug.change(fn=set_debug, inputs=[debug], outputs=[])
+                
+        return [
+            enabled,
+            targets,
+            weight,
+            disable_neg,
+            strong,
+            padding,
+            lerp,
+            debug,
+        ]
+    
+    def process(
+        self,
+        p: StableDiffusionProcessing,
+        enabled: bool,
+        targets_: str,
+        weight: Union[float,int],
+        disable_neg: bool,
+        strong: bool,
+        padding: Union[str,int],
+        intp: str,
+        debug: bool,
+    ):
+        set_debug(debug)
+        
+        if self.last_hooker is not None:
+            self.last_hooker.__exit__(None, None, None)
+            self.last_hooker = None
+        
+        if not enabled:
+            return
+        
+        if targets_ is None or len(targets_) == 0:
+            return
+        
+        targets = [x.strip() for x in targets_.split(',')]
+        targets = [x for x in targets if len(x) != 0]
+        
+        if len(targets) == 0:
+            return
+        
+        if padding is None:
+            padding = PAD
+        elif isinstance(padding, str):
+            if len(padding) == 0:
+                padding = PAD
+            else:
+                try:
+                    padding = int(padding)
+                except:
+                    if not padding.endswith('</w>'):
+                        padding += '</w>'
+        
+        weight = float(weight)
+        intp = intp.lower()
+        
+        self.last_hooker = Hook(
+            enabled=True,
+            targets=targets,
+            padding=padding,
+            weight=weight,
+            disable_neg=disable_neg,
+            strong=strong,
+            interpolate=intp,
+        )
+        
+        self.last_hooker.setup(p)
+        self.last_hooker.__enter__()
+        
+        p.extra_generation_params.update({
+            f'{NAME} enabled': enabled,
+            f'{NAME} targets': targets,
+            f'{NAME} padding': padding,
+            f'{NAME} weight': weight,
+            f'{NAME} disable_for_neg': disable_neg,
+            f'{NAME} strong': strong,
+            f'{NAME} interpolation': intp,
+        })
+
+init_xyz(Script, NAME)

sd-webui-cutoff/scripts/cutofflib/embedding.py

@@ -0,0 +1,214 @@
+from dataclasses import dataclass
+from itertools import product
+import re
+from typing import Union, List, Tuple
+import numpy as np
+import open_clip
+from modules.sd_hijack_clip import FrozenCLIPEmbedderWithCustomWordsBase as CLIP
+from modules import prompt_parser, shared
+from scripts.cutofflib.utils import log
+
+class ClipWrapper:
+    def __init__(self, te: CLIP):
+        self.te = te
+        self.v1 = hasattr(te.wrapped, 'tokenizer')
+        self.t = (
+            te.wrapped.tokenizer if self.v1
+            else open_clip.tokenizer._tokenizer
+        )
+    
+    def token_to_id(self, token: str) -> int:
+        if self.v1:
+            return self.t._convert_token_to_id(token) # type: ignore
+        else:
+            return self.t.encoder[token]
+    
+    def id_to_token(self, id: int) -> str:
+        if self.v1:
+            return self.t.convert_ids_to_tokens(id) # type: ignore
+        else:
+            return self.t.decoder[id]
+    
+    def ids_to_tokens(self, ids: List[int]) -> List[str]:
+        if self.v1:
+            return self.t.convert_ids_to_tokens(ids) # type: ignore
+        else:
+            return [self.t.decoder[id] for id in ids]
+    
+    def token(self, token: Union[int,str]):
+        if isinstance(token, int):
+            return Token(token, self.id_to_token(token))
+        else:
+            return Token(self.token_to_id(token), token)
+
+
+@dataclass
+class Token:
+    id: int
+    token: str
+
+class CutoffPrompt:
+    
+    @staticmethod
+    def _cutoff(prompt: str, clip: CLIP, tokens: List[str], padding: str):
+        def token_count(text: str):
+            tt = token_to_block(clip, text)
+            # tt[0] == clip.id_start (<|startoftext|>)
+            for index, (t, _) in enumerate(tt):
+                if t.id == clip.id_end: # <|endoftext|>
+                    return index - 1
+            return 0 # must not happen...
+        
+        re_targets = [ re.compile(r'\b' + re.escape(x) + r'\b') for x in tokens ]
+        replacer = [ ' ' + ' '.join([padding] * token_count(x)) + ' ' for x in tokens ]
+        
+        rows: List[Tuple[str,str]] = []
+        for block in prompt.split(','):
+            b0 = block
+            for r, p in zip(re_targets, replacer):
+                block = r.sub(p, block)
+            b1 = block
+            rows.append((b0, b1))
+        
+        return rows
+    
+    def __init__(self, prompt: str, clip: CLIP, tokens: List[str], padding: str):
+        self.prompt = prompt
+        rows = CutoffPrompt._cutoff(prompt, clip, tokens, padding)
+        self.base = np.array([x[0] for x in rows])
+        self.cut  = np.array([x[1] for x in rows])
+        self.sw = np.array([False] * len(rows))
+    
+    @property
+    def block_count(self):
+        return self.base.shape[0]
+    
+    def switch(self, block_index: int, to: Union[bool,None] = None):
+        if to is None:
+            to = not self.sw[block_index]
+        self.sw[block_index] = to
+        return to
+    
+    def text(self, sw=None):
+        if sw is None:
+            sw = self.sw
+        blocks = np.where(sw, self.cut, self.base)
+        return ','.join(blocks)
+    
+    def active_blocks(self) -> np.ndarray:
+        indices, = (self.base != self.cut).nonzero()
+        return indices
+    
+    def generate(self):
+        indices = self.active_blocks()
+        for diff_sw in product([False, True], repeat=indices.shape[0]):
+            sw = np.full_like(self.sw, False)
+            sw[indices] = diff_sw
+            yield diff_sw, self.text(sw)
+
+
+def generate_prompts(
+    clip: CLIP,
+    prompt: str,
+    targets: List[str],
+    padding: Union[str,int,Token],
+) -> CutoffPrompt:
+    
+    te = ClipWrapper(clip)
+    
+    if not isinstance(padding, Token):
+        o_pad = padding
+        padding = te.token(padding)
+        if padding.id == clip.id_end:
+            raise ValueError(f'`{o_pad}` is not a valid token.')
+    
+    result = CutoffPrompt(prompt, clip, targets, padding.token.replace('</w>', ''))
+    
+    log(f'[Cutoff] replace: {", ".join(targets)}')
+    log(f'[Cutoff] to: {padding.token} ({padding.id})')
+    log(f'[Cutoff] original: {prompt}')
+    for i, (_, pp) in enumerate(result.generate()):
+        log(f'[Cutoff]       #{i}: {pp}')
+    
+    return result
+
+
+def token_to_block(clip: CLIP, prompt: str):
+    te = ClipWrapper(clip)
+    
+    # cf. sd_hijack_clip.py
+    
+    parsed = prompt_parser.parse_prompt_attention(prompt)
+    tokenized: List[List[int]] = clip.tokenize([text for text, _ in parsed])
+    
+    CHUNK_LENGTH = 75
+    id_start = te.token(clip.id_start) # type: ignore
+    id_end = te.token(clip.id_end) # type: ignore
+    comma = te.token(',</w>')
+    
+    last_comma = -1
+    current_block = 0
+    current_tokens: List[Tuple[Token,int]] = []
+    result: List[Tuple[Token,int]] = []
+    
+    def next_chunk():
+        nonlocal current_tokens, last_comma
+        
+        to_add = CHUNK_LENGTH - len(current_tokens)
+        if 0 < to_add:
+            current_tokens += [(id_end, -1)] * to_add
+        
+        current_tokens = [(id_start, -1)] + current_tokens + [(id_end, -1)]
+        
+        last_comma = -1
+        result.extend(current_tokens)
+        current_tokens = []
+        
+    for tokens, (text, weight) in zip(tokenized, parsed):
+        if text == 'BREAK' and weight == -1:
+            next_chunk()
+            continue
+        
+        p = 0
+        while p < len(tokens):
+            token = tokens[p]
+            
+            if token == comma.id:
+                last_comma = len(current_tokens)
+                current_block += 1
+            
+            elif (
+                shared.opts.comma_padding_backtrack != 0 
+                and len(current_tokens) == CHUNK_LENGTH
+                and last_comma != -1 
+                and len(current_tokens) - last_comma <= shared.opts.comma_padding_backtrack
+            ):
+                break_location = last_comma + 1
+                reloc_tokens = current_tokens[break_location:]
+                current_tokens = current_tokens[:break_location]
+                next_chunk()
+                current_tokens = reloc_tokens
+            
+            if len(current_tokens) == CHUNK_LENGTH:
+                next_chunk()
+            
+            embedding, embedding_length_in_tokens = clip.hijack.embedding_db.find_embedding_at_position(tokens, p)
+            if embedding is None:
+                if token == comma.id:
+                    current_tokens.append((te.token(token), -1))
+                else:
+                    current_tokens.append((te.token(token), current_block))
+                p += 1
+                continue
+
+            emb_len = int(embedding.vec.shape[0])
+            if len(current_tokens) + emb_len > CHUNK_LENGTH:
+                next_chunk()
+
+            current_tokens += [(te.token(0), current_block)] * emb_len
+            p += embedding_length_in_tokens
+            
+    if len(current_tokens) > 0:
+        next_chunk()
+    
+    return result