Title: Personalized Vision via Visual In-Context Learning
Venue: NeurIPS 2025
Accepted: no
Keywords: personalization

visual in-context learning

diffusion models
Abstract: Modern vision models, trained on large-scale annotated datasets, excel at predefined tasks such as segmentation but struggle to adapt flexibly to personalized vision tasks—tasks defined at test-time by users with customized objects or novel objectives.
Existing personalization approaches typically rely on synthesizing additional training data or fine-tuning the entire model, limiting flexibility and incurring significant computational cost.
Inspired by recent advances in natural language processing, we explore a new direction: leveraging visual generative models for personalized vision via in-context learning.
We introduce a structured four-panel input format, where a single annotated example specifies the personalized visual task, allowing the model to interpret and generalize the task to new inputs without further fine-tuning.
To enable this one-shot capability, we construct a Visual-Relation tuning dataset tailored to personalized vision in-context learning.
Extensive experiments demonstrate that our approach (i) surpasses fine-tuning and synthetic-data baselines on personalized segmentation, (ii) enables test-time definition of novel personalized tasks, and (iii) generalizes across both visual recognition and generation settings.
Our work establishes a new paradigm for personalized vision, combining the adaptability of in-context learning with the visual reasoning capabilities of generative models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Visual In-Context Learning method, called PICO, by creating a four-panel input format {A, A', B, B'}, prompting the model to infer the transformation pattern from A to A' and apply it to generate B to B'. The authors constructed a dataset called VisRel in this format, encompassing various visual tasks, including perception, low-level vision, and generation tasks. They then fine-tuned FLUX.1 using LoRA on the constructed dataset to achieve the above objective.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The paper focuses on an important problem: Visual In-Context Learning, and demonstrates several interesting effects that are challenging to achieve with fixed task formats, such as \"Stylize only the background\" and \"Detect only the uppermost contours\" shown in Fig. 1.\n\n2. The implementation of the proposed method is highly lightweight, utilizing only a single GPU and a training set consisting of only 315 samples.\n\nWeakness:\n\n1. The method is relatively simple and lacks technical contribution. The task, Visual In-Context Learning, was already defined by works such as Painter, LVM. Collecting datasets of low-level vision, perception, and generation, into a unified format for training has also been achieved by Painter and InstructDiffusion, etc. Moreover, using multiple images concatenated as input to the DiT model for Visual In-Context Learning aligns with recent research trends, such as IC-LoRA. This work lacks unique contributions in either task objective or technical details.\n\n2. For dataset construction, several key details lack sufficient discussion. For example:\n\n- How are the colors of masks for different categories or instances defined in segmentation tasks, and do they significantly affect the results? \n\n- Why were the current 27 tasks chosen for dataset construction? While the paper includes an ablation study on the number of tasks, given the significant differences among different tasks, the impact of task types should have a greater focus. For instance, what advantages do low-level vision tasks bring to achieving personalized segmentation or generation? Experiments and discussions are needed.\n\n3. The writing of this paper is suboptimal, with some important details being unclear or overly scattered across different sections:\n\n- About the dataset: Section 3.1 does not mention the dataset scale at all, only states that it contains \"more than 25 visual tasks.\" Section 4.1 describes that the dataset contains 315 samples and 27 tasks. However, the specific tasks are still not provided, and it can only be found in the supplementary material. Information about the dataset size and the tasks it includes should be provided in the method section, to better aid readers in understanding the paper.\n\n- About the model: Are the input four images concatenated into a single image before being passed through the VAE encoder, or are they encoded separately and then concatenated in the latent space? The writing in Section 3.2 is quite confusing. For instance, on line 140, the authors state that \"X is a noisy placeholder,\" but on line 150, they mention \"applying noise to x_0 = E(X) to obtain x_t.\" This is confusing, as latent diffusion models do not sample a noise in the pixel space and then encode it.\n\n4. As the main experimental results of the paper, the proposed method PICO's metrics in Table. 1 are not particularly high. PICO achieves the best results in only two metrics and shows a significant gap with SegGPT in some others.\n\nQUESTIONS:\nPlease refer to the Weakness section above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

comments: SUMMARY:
This paper proposes PICO, a novel framework for personalized vision by reformulating it as a visual in-context learning (ICL) problem. Instead of relying on task-specific fine-tuning or synthetic data, the authors introduce a unified visual relation space that allows pretrained diffusion transformers to perform user-defined visual tasks based on a single visual example. PICO achieves strong performance and flexibility across various personalized tasks, demonstrating its potential for real-world applications and establishing diffusion models as effective visual in-context learners.

STRENGTHS AND WEAKNESSES:
Strengths
- The paper is clearly structured and easy to follow.

- The experimental section is thorough, with extensive evaluations across diverse tasks.

- The research direction is meaningful and timely, addressing the growing need for flexible, personalized vision systems.

Weaknesses：
- Since the VisRel dataset and the PICO implementation code are not publicly available at this stage, it is difficult to fully assess the soundness of the dataset construction and the practical validity of the proposed method.

QUESTIONS:
- How is the VisRel dataset constructed? Does it contain the same object categories or task types as those seen during test time? If so, how might this overlap affect the evaluation of the model?

- If a new task, not included in the original 25 tasks of VisRel, is introduced, how should the model be adapted? Does this require full retraining?

- In Figure 3, the noise placeholder “X” appears to be positioned in the bottom-right corner by default. Is this placement fixed across training and inference? If so, does the spatial position of the placeholder affect the model’s in-context reasoning performance?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies \"personalization\" of vision models using in-context learning. The method trains a model to map from A,A',B --> B', where A and A' are a pair of images that exemplifies a target transformation, B is a query image and B' is, ideally, the outcome of applying the A-->A' transformation to B. This is setup as an inpainting problem where all the images are arranged in a grid with the B' image masked out. A diffusion model is used to solve the inpainting, initializing B' as noise and denoising to complete the analogy. This model is trained on a novel dataset that shows a diverse range of image transformations. Experiments show that performs well on a wide range of tasks, notably outperforming several prior approaches to personalization.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* Simple approach\n* Good results\n* The model and dataset could serve as useful tools upon release (code and data release is promised)\n* The ablations provide some information about what works and what doesn't\n\nWeaknesses:\n* The idea is essentially the same as Bar et al.'s Visual Prompting: train a model to map from A,A',B --> B', formulated as inpainting B', and train it on a diverse set of transformations. This approach was further studied in Analogist (Gu et al. 2024), where a diffusion model served as a prior, like in the current paper. A difference with Analogist is the current paper finetunes on dataset, which Bar et al. did as well but Analogist did not.\n* Given the above, the paper oversells its novelty, in my opinion. For example, the abstract claims this paper \"establishes a new paradigm for personalized vision,\" yet how exactly is this paradigm different from that in Bar et al. or Analogist, among others? The writing is peppered with subjective language that praises the method/paper but does not substantively inform the reader (Examples: \"_extensive_ experiments\", \"_unprecedented_ flexibility\", \"_promising_ results\", emphasis mine).\n* It's not clear to me what are the important new ideas or discoveries I should take from this paper, that are not present in the prior work.\n* Most of the experiments (aside from the ablations) are apples-to-oranges: different architectures, trained on different datasets, with different pretraining, different conditioning, etc.\n\nI think this paper can make a strong contribution as a \"simple baseline\" type paper. It shows great results on personalization benchmarks using a paradigm that, arguably, precedes more recent work in this area and is simpler, leveraging standard tools (finetuning a diffusion model, steering a model via in-context examples). However, I think reframing the paper in this way will require a thorough rewrite that may be beyond what is reasonable in a rebuttal.\n\nQUESTIONS:\n1. What is the precise novelty compared to Bar et al. and Analogist?\n2. What new insights should I be taking away?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the personalized vision setting, in which in test time users can flexible define the task via visual in-context learning.\nFor example, user might want to give the task \"segment my dog\" by given example A -> A', then given a test image B, proposed model (called PICO) will generate B' image.\nPICO was built upon FLUX model, which is trained to denoise the quad-grid [A, A', B, B']. Experiments results show the promising of this paradigm for personalized vision tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- On a high level idea, I appreciate the \"personalized vision\" setting, in which give users flexibility in test time to define the task.\n- The paper is generally well-written and easy to follow.\n- Experiments section cover not all, but sufficient tasks that cover generally a wide range of vision tasks (e.g., segmentation, doodling)\n\nWeaknesses:\n- The proposed grid-like images are very similar to the Visual Prompting via Image Inpainting (NeurIPS 2023). While this paper (PICO) did cite Visual Prompting, a further explanation how different is Visual Prompting vs. PICO is needed. In another words, is it true that PICO is only (i) trained with task-specific and (ii) use better base model (i.e., FLUX) compared to Visual Prompting?\n- It is necessary to compare PICO to OminiGen (CVPR 2025) (https://arxiv.org/abs/2409.11340) and Large Vision Models (CVPR 2024) (https://arxiv.org/abs/2312.00785). Given that these models are also (i) allow free-form guidance (e.g., either via text or in-context inputs), (ii) trained on large datasets that cover a wide range of tasks (i.e., similar to proposed VisRel dataset).\n\nQUESTIONS:\nMy current rating is Borderline reject, as my main concern are: (i) clarify the different to Visual Prompting and (ii) lack of mention to highly relevant works.\nI'm happy to raise my rating if the rebuttal answer address my concerns.\n\nMinor question: Will the grid-size 512 (each image 256x256) will be the limitation of this approach?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}