Package: scepter
🪄SCEPTER📖 Table of ContentsIntroductionNewsInstallationGetting StartedSCEPTER StudioGalleryFeaturesLearn MoreLicense📝 IntroductionSCEPTER is an open-source code repository dedicated to generative training, fine-tuning, and inference, encompassing a suite of downstream tasks such as image generation, transfer, editing. It integrates popular community-driven implementations as well as proprietary methods by Tongyi Lab of Alibaba Group, offering a comprehensive toolkit for researchers and practitioners in the field of AIGC. This versatile library is designed to facilitate innovation and accelerate development in the rapidly evolving domain of generative models.Main Feature:Task:Text-to-image generationControllable image synthesisImage editing (TODO)Training / Inference:Distribute: DDP / FSDP / FairScale / XformersFile system: Local / Http / OSS / ModelscopeDeploy:Data managementTrainingInferenceCurrently supported approaches (and counting):SD Series:Stable Diffusion v1.5/Stable Diffusion v2.1/Stable Diffusion XLSCEdit:SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection EditingRes-Tuning(TODO):Res-Tuning: A Flexible and Efficient Tuning Paradigm via Unbinding Tuner from Backbone🎉 News[2024.02]: We release new SCEdit controllable image synthesis models for SD v2.1 and SD XL. Multiple strategies applied to accelerate inference time for SCEPTER Studio.[2024.01]: We releaseSCEPTER Studio, an integrated toolkit for data management, model training and inference based onGradio.[2024.01]:SCEditsupport controllable image synthesis for training and inference.[2023.12]: We proposeSCEdit, an efficient and controllable generation framework.[2023.12]: We release🪄SCEPTERlibrary.🛠️ InstallationCreate new environmentcondaenvcreate-fenvironment.yaml
condaactivatescepterWe recommend installing the specific version of PyTorch and accelerate toolboxxFormers. You can install these recommended version by pip:pipinstall-rrequirements/recommended.txtInstall SCEPTER by thepipcommand:pipinstallscepter🚀 Getting StartedDatasetModelscope FormatWe use acustom-stylized dataset, which included classes 3D, anime, flat illustration, oil painting, sketch, and watercolor, each with 30 image-text pairs.# pip install modelscopefrommodelscope.msdatasetsimportMsDatasetms_train_dataset=MsDataset.load('style_custom_dataset',namespace='damo',subset_name='3D',split='train_short')print(next(iter(ms_train_dataset)))CSV FormatFor the data format used by SCEPTER Studio, please refer to3D_example_csv.zip.TXT FormatTo facilitate starting training in command-line mode, you can use a dataset in text format, please refer to3D_example_txt.zipmkdir-pcache/dataset/&&wget'https://modelscope.cn/api/v1/models/damo/scepter_scedit/repo?Revision=master&FilePath=dataset/3D_example_txt.zip'-Ocache/dataset/3D_example_txt.zip&&unzipcache/dataset/3D_example_txt.zip-dcache/dataset/&&rmcache/dataset/3D_example_txt.zipTrainingWe provide a framework for training and inference, so the script below is just for illustration purposes. To achieve better results, you can modify the corresponding parameters as needed.Text-to-Image GenerationSCEditpythonscepter/tools/run_train.py--cfgscepter/methods/scedit/t2i/sd15_512_sce_t2i.yaml# SD v1.5pythonscepter/tools/run_train.py--cfgscepter/methods/scedit/t2i/sd21_768_sce_t2i.yaml# SD v2.1pythonscepter/tools/run_train.py--cfgscepter/methods/scedit/t2i/sdxl_1024_sce_t2i.yaml# SD XLExisting Tuning Strategiespythonscepter/tools/run_train.py--cfgscepter/methods/examples/generation/stable_diffusion_1.5_512.yaml# fully-tuning on SD v1.5pythonscepter/tools/run_train.py--cfgscepter/methods/examples/generation/stable_diffusion_2.1_768_lora.yaml# lora-tuning on SD v2.1Data Text Format# Download the 3D_example_txt.zip as previously mentionedpythonscepter/tools/run_train.py--cfgscepter/methods/scedit/t2i/sdxl_1024_sce_t2i_datatxt.yamlControllable Image SynthesisSCEditThe YAML configuration can be modified to combine different base models and conditions. The following is provided as an example.pythonscepter/tools/run_train.py--cfgscepter/methods/scedit/ctr/sd15_512_sce_ctr_hed.yaml# SD v1.5 + hedpythonscepter/tools/run_train.py--cfgscepter/methods/scedit/ctr/sd21_768_sce_ctr_canny.yaml# SD v2.1 + cannypythonscepter/tools/run_train.py--cfgscepter/methods/scedit/ctr/sd21_768_sce_ctr_pose.yaml# SD v2.1 + posepythonscepter/tools/run_train.py--cfgscepter/methods/scedit/ctr/sdxl_1024_sce_ctr_depth.yaml# SD XL + depthpythonscepter/tools/run_train.py--cfgscepter/methods/scedit/ctr/sdxl_1024_sce_ctr_color.yaml# SD XL + colorData Text Format# Download the 3D_example_txt.zip as previously mentionedpythonscepter/tools/run_train.py--cfgscepter/methods/scedit/ctr/sdxl_1024_sce_ctr_color_datatxt.yamlInferenceBase Model Inferencepythonscepter/tools/run_inference.py--cfgscepter/methods/examples/generation/stable_diffusion_1.5_512.yaml--prompt'a cute dog'--save_folder'inference'# generation on SD v1.5pythonscepter/tools/run_inference.py--cfgscepter/methods/examples/generation/stable_diffusion_2.1_768.yaml--prompt'a cute dog'--save_folder'inference'# generation on SD v2.1pythonscepter/tools/run_inference.py--cfgscepter/methods/examples/generation/stable_diffusion_xl_1024.yaml--prompt'a cute dog'--save_folder'inference'# generation on SD XLFine-tuned Model Inferencepythonscepter/tools/run_inference.py--cfgscepter/methods/scedit/t2i/sd15_512_sce_t2i_swift.yaml--pretrained_model'cache/save_data/sd15_512_sce_t2i_swift/checkpoints/ldm_step-100.pth'--prompt'A close up of a small rabbit wearing a hat and scarf'--save_folder'trained_test_prompt_rabbit'Controllable Image Synthesis InferenceSCEditpythonscepter/tools/run_inference.py--cfgscepter/methods/scedit/ctr/sd21_768_sce_ctr_canny.yaml--num_samples1--prompt'a single flower is shown in front of a tree'--save_folder'test_flower_canny'--image_size768--taskcontrol--image'asset/images/flower.jpg'--control_modecanny--pretrained_modelms://damo/scepter_scedit@controllable_model/SD2.1/canny_control/0_SwiftSCETuning/pytorch_model.bin# cannypythonscepter/tools/run_inference.py--cfgscepter/methods/scedit/ctr/sd21_768_sce_ctr_pose.yaml--num_samples1--prompt'super mario'--save_folder'test_mario_pose'--image_size768--taskcontrol--image'asset/images/pose_source.png'--control_modesource--pretrained_modelms://damo/scepter_scedit@controllable_model/SD2.1/pose_control/0_SwiftSCETuning/pytorch_model.bin# pose🖥️ SCEPTER StudioLaunchTo fully experienceSCEPTER Studio, you can launch the following command line:pipinstallscepter
python-mscepter.tools.webuior run after clone repo codegitclonehttps://github.com/modelscope/scepter.gitPYTHONPATH=.pythonscepter/tools/webui.py--cfgscepter/methods/studio/scepter_ui.yamlThe startup ofSCEPTER Studioeliminates the need for manual downloading and organizing of models; it will automatically load the corresponding models and store them in a local directory.
Depending on the network and hardware situation, the initial startup usually requires 15-60 minutes, primarily involving the download and processing of SDv1.5, SDv2.1, and SDXL models.
Therefore, subsequent startups will become much faster (about one minute) as downloading is no longer required.Modelscope StudioWe deploy a work studio on Modelscope that includes only the inference tab, please refer toms_scepter_studio🖼️ GalleryDragon Year Special: Dragon TunerGold Dragon TunerSloppy Dragon TunerRed Dragon Tuner+ Papercraft MantraAzure Dragon Tuner+ Pose ControlText Effect ImageConditional ImageMidas Control"Race track, top view"Midas Control+ Watercolor Mantra"white lilies"Midas Control+ Dragon Tuner"Spring Festival, Chinese dragon"✨ FeaturesText-to-Image GenerationModelSCEditFullLoRASD 1.5🪄✅✅SD 2.1🪄✅✅SD XL🪄✅✅Controllable Image SynthesisSCEditModelCannyHEDDepthPoseColorSD 1.5✅✅✅✅✅SD 2.1🪄🪄🪄🪄🪄SD XL🪄🪄🪄🪄🪄Model URL✅ indicates support for both training and inference.🪄 denotes that the model has been published.More models will be released in the future.ModelURLSCEditModelScopeHuggingFacePS: Scripts running within the SCEPTER framework will automatically fetch and load models based on the required dependency files, eliminating the need for manual downloads.🔍 Learn MoreAlibaba TongYi Vision Intelligence LabDiscover more about open-source projects on image generation, video generation, and editing tasks.ModelScope libraryModelScope Library is the model library of ModelScope project, which contains a large number of popular models.SWIFT librarySWIFT (Scalable lightWeight Infrastructure for Fine-Tuning) is an extensible framwork designed to faciliate lightweight model fine-tuning and inference.LicenseThis project is licensed under theApache License (Version 2.0).