---
license: mit
---
Project page: https://gpt4vision.github.io/gpt4sgg/

GPT4SGG aims to synthesize a scene graph from global & localized narratives for an image. 
The model is fine-tuned on instruction-following data generated by GPT-4.


Usage demo:
```
import json

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig

tokenizer = AutoTokenizer.from_pretrained("JosephZ/gpt4sgg-llama2-13b-int8", trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained("JosephZ/gpt4sgg-llama2-13b-int8", trust_remote_code=True, 
                                             load_in_8bit=True,
                                             device_map="cuda:0")

custom_input = [
    {"image_id": "101115", "width": 640, "height": 480, 
     "objects": ["person.1:[344, 374, 449, 480]", "person.2:[523, 274, 602, 479]", "person.3:[593, 366, 636, 474]",
                 "kite.4:[230, 85, 313, 137]", "kite.5:[461, 394, 529, 468]", "person.6:[456, 345, 512, 474]"], 
     "captions": {"global": "a group of people flying a kite in a field", 
                  "Union(person.2:[523, 274, 602, 479], person.3:[593, 366, 636, 474]) ; Union(person.2:[523, 274, 602, 479], kite.5:[461, 394, 529, 468])": "a woman in a dress", 
                  "Union(kite.5:[461, 394, 529, 468], person.6:[456, 345, 512, 474])": "a man with no shirt"}
     }]
custom_input = json.dumps(custom_input) # to str

prompt  = """[INST] <<SYS>>
You are a helpful AI visual assistant. Now, you are seeing image data. Each image provides a set of objects, and a set of captions for global and localized descriptions. 
<</SYS>> Extract relationship triplets from image data, each characterized by a unique "image_id", image dimensions,  a set of objects consisting of categories (formatted as "[category].[number]") and bounding boxes (in "xyxy" format). Each image data includes a global description for the entire image and localized descriptions for specific regions (notated as "Union(name1:box1, name2:box2)", keys with ";" in captions like "Union(name1:box1, name2:box2); Union(name3:box3, name4:box4)" refer to multiple union regions share the same caption).                
Here are the requirements for the task:
1. Process each image individually: Focus on one image at a time and give a comprehensive output for that specific image before moving to the next.
2. Infer interactions and spatial relationships: Utilize objects' information and both global and localized descriptions to determine relationships between objects(e.g., "next to", "holding", "held by", etc.).
3. Maintain logical consistency: Avoid impossible or nonsensical relationships (e.g., a person cannot be riding two different objects simultaneously, a tie cannot be worn by two persons, etc.).
4. Eliminate duplicate entries: Each triplet in the output must be unique and non-repetitive.
5. Output should be formatted as a list of dicts in JSON format, containing "image_id" and "relationships" for each image.
                
Example output: \n```
[
{"image_id": "123456",
 "relationships": [
    {"source": "person.1", "target": "skateboard.2", "relation": "riding"},
    {"source": "person.4", "target": "shirt.3", "relation": "wearing"},
    {"source": "person.2", "target": "bottle.5", "relation": "holding"},
    {"source": "person.4", "target": "bus.1", "relation": "near"},
  ]         
},
{"image_id": "23455",
 "relationships": [ 
    {"source": "man.1", "target": "car.1", "relation": "driving"}
  ]         
}           
]\n```             

Ensure that each image's data is processed and outputted separately to maintain clarity and accuracy in the relationship analysis.
                           
### Input:\n```\nPLACE_HOLDER\n```
[/INST]
### Output:\n```
"""

prompt = prompt.replace("PLACE_HOLDER", custom_input)



gen_cfg = GenerationConfig(do_sample=True, temperature=0.7, top_p=0.95)

prompts = [prompt] # batch prompts
inputs = tokenizer(prompts, padding=True, return_tensors='pt')
outputs = model.generate(input_ids=inputs['input_ids'].cuda(),
                         generation_config=gen_cfg,
                         return_dict_in_generate=True,
                         max_length=4096, 
                         pad_token_id=tokenizer.eos_token_id,
                         )                          
outputs_text = tokenizer.batch_decode(outputs.sequences, skip_special_tokens=True)
outputs_text = [e[len(prompt):] for e, prompt in zip(outputs_text, prompts)]

print("*"*10, " prompt:", prompt)
print("*"*10, " resp:", outputs_text[0])
```