ChineseTextModel: Weight loading error
Author: ranck626
Pull request: no
Labels: name: bug

### System Info

Some weights of the model checkpoint at OFA-Sys/chinese-clip-vit-base-patch16 were not used when initializing ChineseCLIPTextModel: ['logit_scale', 'text_model.embeddings.LayerNorm.bias', 'text_model.embeddings.LayerNorm.weight', 'text_model.embeddings.position_embeddings.weight', 'text_model.embeddings.position_ids', 'text_model.embeddings.token_type_embeddings.weight', 'text_model.embeddings.word_embeddings.weight', 'text_model.encoder.layer.0.attention.output.LayerNorm.bias', 'text_model.encoder.layer.0.attention.output.LayerNorm.weight', 'text_model.encoder.layer.0.attention.output.dense.bias', 'text_model.encoder.layer.0.attention.output.dense.weight', 'text_model.encoder.layer.0.attention.self.key.bias', 'text_model.encoder.layer.0.attention.self.key.weight', 'text_model.encoder.layer.0.attention.self.query.bias', 'text_model.encoder.layer.0.attention.self.query.weight', 'text_model.encoder.layer.0.attention.self.value.bias', 'text_model.encoder.layer.0.attention.self.value.weight', 'text_model.encoder.layer.0.intermediate.dense.bias', 'text_model.encoder.layer.0.intermediate.dense.weight', 'text_model.encoder.layer.0.output.LayerNorm.bias', 'text_model.encoder.layer.0.output.LayerNorm.weight', 'text_model.encoder.layer.0.output.dense.bias', 'text_model.encoder.layer.0.output.dense.weight', 'text_model.encoder.layer.1.attention.output.LayerNorm.bias', 'text_model.encoder.layer.1.attention.output.LayerNorm.weight', 'text_model.encoder.layer.1.attention.output.dense.bias', 'text_model.encoder.layer.1.attention.output.dense.weight', 'text_model.encoder.layer.1.attention.self.key.bias', 'text_model.encoder.layer.1.attention.self.key.weight', 'text_model.encoder.layer.1.attention.self.query.bias', 'text_model.encoder.layer.1.attention.self.query.weight', 'text_model.encoder.layer.1.attention.self.value.bias', 'text_model.encoder.layer.1.attention.self.value.weight', 'text_model.encoder.layer.1.intermediate.dense.bias', 'text_model.encoder.layer.1.intermediate.dense.weight', 'text_model.encoder.layer.1.output.LayerNorm.bias', 'text_model.encoder.layer.1.output.LayerNorm.weight', 'text_model.encoder.layer.1.output.dense.bias', 'text_model.encoder.layer.1.output.dense.weight', 'text_model.encoder.layer.10.attention.output.LayerNorm.bias', 'text_model.encoder.layer.10.attention.output.LayerNorm.weight', 'text_model.encoder.layer.10.attention.output.dense.bias', 'text_model.encoder.layer.10.attention.output.dense.weight', 'text_model.encoder.layer.10.attention.self.key.bias', 'text_model.encoder.layer.10.attention.self.key.weight', 'text_model.encoder.layer.10.attention.self.query.bias', 'text_model.encoder.layer.10.attention.self.query.weight', 'text_model.encoder.layer.10.attention.self.value.bias', 'text_model.encoder.layer.10.attention.self.value.weight', 'text_model.encoder.layer.10.intermediate.dense.bias', 'text_model.encoder.layer.10.intermediate.dense.weight', 'text_model.encoder.layer.10.output.LayerNorm.bias', 'text_model.encoder.layer.10.output.LayerNorm.weight', 'text_model.encoder.layer.10.output.dense.bias', 'text_model.encoder.layer.10.output.dense.weight', 'text_model.encoder.layer.11.attention.output.LayerNorm.bias', 'text_model.encoder.layer.11.attention.output.LayerNorm.weight', 'text_model.encoder.layer.11.attention.output.dense.bias', 'text_model.encoder.layer.11.attention.output.dense.weight', 'text_model.encoder.layer.11.attention.self.key.bias', 'text_model.encoder.layer.11.attention.self.key.weight', 'text_model.encoder.layer.11.attention.self.query.bias', 'text_model.encoder.layer.11.attention.self.query.weight', 'text_model.encoder.layer.11.attention.self.value.bias', 'text_model.encoder.layer.11.attention.self.value.weight', 'text_model.encoder.layer.11.intermediate.dense.bias', 'text_model.encoder.layer.11.intermediate.dense.weight', 'text_model.encoder.layer.11.output.LayerNorm.bias', 'text_model.encoder.layer.11.output.LayerNorm.weight', 'text_model.encoder.layer.11.output.dense.bias', 'text_model.encoder.layer.11.output.dense.weight', 'text_model.encoder.layer.2.attention.output.LayerNorm.bias', 'text_model.encoder.layer.2.attention.output.LayerNorm.weight', 'text_model.encoder.layer.2.attention.output.dense.bias', 'text_model.encoder.layer.2.attention.output.dense.weight', 'text_model.encoder.layer.2.attention.self.key.bias', 'text_model.encoder.layer.2.attention.self.key.weight', 'text_model.encoder.layer.2.attention.self.query.bias', 'text_model.encoder.layer.2.attention.self.query.weight', 'text_model.encoder.layer.2.attention.self.value.bias', 'text_model.encoder.layer.2.attention.self.value.weight', 'text_model.encoder.layer.2.intermediate.dense.bias', 'text_model.encoder.layer.2.intermediate.dense.weight', 'text_model.encoder.layer.2.output.LayerNorm.bias', 'text_model.encoder.layer.2.output.LayerNorm.weight', 'text_model.encoder.layer.2.output.dense.bias', 'text_model.encoder.layer.2.output.dense.weight', 'text_model.encoder.layer.3.attention.output.LayerNorm.bias', 'text_model.encoder.layer.3.attention.output.LayerNorm.weight', 'text_model.encoder.layer.3.attention.output.dense.bias', 'text_model.encoder.layer.3.attention.output.dense.weight', 'text_model.encoder.layer.3.attention.self.key.bias', 'text_model.encoder.layer.3.attention.self.key.weight', 'text_model.encoder.layer.3.attention.self.query.bias', 'text_model.encoder.layer.3.attention.self.query.weight', 'text_model.encoder.layer.3.attention.self.value.bias', 'text_model.encoder.layer.3.attention.self.value.weight', 'text_model.encoder.layer.3.intermediate.dense.bias', 'text_model.encoder.layer.3.intermediate.dense.weight', 'text_model.encoder.layer.3.output.LayerNorm.bias', 'text_model.encoder.layer.3.output.LayerNorm.weight', 'text_model.encoder.layer.3.output.dense.bias', 'text_model.encoder.layer.3.output.dense.weight', 'text_model.encoder.layer.4.attention.output.LayerNorm.bias', 'text_model.encoder.layer.4.attention.output.LayerNorm.weight', 'text_model.encoder.layer.4.attention.output.dense.bias', 'text_model.encoder.layer.4.attention.output.dense.weight', 'text_model.encoder.layer.4.attention.self.key.bias', 'text_model.encoder.layer.4.attention.self.key.weight', 'text_model.encoder.layer.4.attention.self.query.bias', 'text_model.encoder.layer.4.attention.self.query.weight', 'text_model.encoder.layer.4.attention.self.value.bias', 'text_model.encoder.layer.4.attention.self.value.weight', 'text_model.encoder.layer.4.intermediate.dense.bias', 'text_model.encoder.layer.4.intermediate.dense.weight', 'text_model.encoder.layer.4.output.LayerNorm.bias', 'text_model.encoder.layer.4.output.LayerNorm.weight', 'text_model.encoder.layer.4.output.dense.bias', 'text_model.encoder.layer.4.output.dense.weight', 'text_model.encoder.layer.5.attention.output.LayerNorm.bias', 'text_model.encoder.layer.5.attention.output.LayerNorm.weight', 'text_model.encoder.layer.5.attention.output.dense.bias', 'text_model.encoder.layer.5.attention.output.dense.weight', 'text_model.encoder.layer.5.attention.self.key.bias', 'text_model.encoder.layer.5.attention.self.key.weight', 'text_model.encoder.layer.5.attention.self.query.bias', 'text_model.encoder.layer.5.attention.self.query.weight', 'text_model.encoder.layer.5.attention.self.value.bias', 'text_model.encoder.layer.5.attention.self.value.weight', 'text_model.encoder.layer.5.intermediate.dense.bias', 'text_model.encoder.layer.5.intermediate.dense.weight', 'text_model.encoder.layer.5.output.LayerNorm.bias', 'text_model.encoder.layer.5.output.LayerNorm.weight', 'text_model.encoder.layer.5.output.dense.bias', 'text_model.encoder.layer.5.output.dense.weight', 'text_model.encoder.layer.6.attention.output.LayerNorm.bias', 'text_model.encoder.layer.6.attention.output.LayerNorm.weight', 'text_model.encoder.layer.6.attention.output.dense.bias', 'text_model.encoder.layer.6.attention.output.dense.weight', 'text_model.encoder.layer.6.attention.self.key.bias', 'text_model.encoder.layer.6.attention.self.key.weight', 'text_model.encoder.layer.6.attention.self.query.bias', 'text_model.encoder.layer.6.attention.self.query.weight', 'text_model.encoder.layer.6.attention.self.value.bias', 'text_model.encoder.layer.6.attention.self.value.weight', 'text_model.encoder.layer.6.intermediate.dense.bias', 'text_model.encoder.layer.6.intermediate.dense.weight', 'text_model.encoder.layer.6.output.LayerNorm.bias', 'text_model.encoder.layer.6.output.LayerNorm.weight', 'text_model.encoder.layer.6.output.dense.bias', 'text_model.encoder.layer.6.output.dense.weight', 'text_model.encoder.layer.7.attention.output.LayerNorm.bias', 'text_model.encoder.layer.7.attention.output.LayerNorm.weight', 'text_model.encoder.layer.7.attention.output.dense.bias', 'text_model.encoder.layer.7.attention.output.dense.weight', 'text_model.encoder.layer.7.attention.self.key.bias', 'text_model.encoder.layer.7.attention.self.key.weight', 'text_model.encoder.layer.7.attention.self.query.bias', 'text_model.encoder.layer.7.attention.self.query.weight', 'text_model.encoder.layer.7.attention.self.value.bias', 'text_model.encoder.layer.7.attention.self.value.weight', 'text_model.encoder.layer.7.intermediate.dense.bias', 'text_model.encoder.layer.7.intermediate.dense.weight', 'text_model.encoder.layer.7.output.LayerNorm.bias', 'text_model.encoder.layer.7.output.LayerNorm.weight', 'text_model.encoder.layer.7.output.dense.bias', 'text_model.encoder.layer.7.output.dense.weight', 'text_model.encoder.layer.8.attention.output.LayerNorm.bias', 'text_model.encoder.layer.8.attention.output.LayerNorm.weight', 'text_model.encoder.layer.8.attention.output.dense.bias', 'text_model.encoder.layer.8.attention.output.dense.weight', 'text_model.encoder.layer.8.attention.self.key.bias', 'text_model.encoder.layer.8.attention.self.key.weight', 'text_model.encoder.layer.8.attention.self.query.bias', 'text_model.encoder.layer.8.attention.self.query.weight', 'text_model.encoder.layer.8.attention.self.value.bias', 'text_model.encoder.layer.8.attention.self.value.weight', 'text_model.encoder.layer.8.intermediate.dense.bias', 'text_model.encoder.layer.8.intermediate.dense.weight', 'text_model.encoder.layer.8.output.LayerNorm.bias', 'text_model.encoder.layer.8.output.LayerNorm.weight', 'text_model.encoder.layer.8.output.dense.bias', 'text_model.encoder.layer.8.output.dense.weight', 'text_model.encoder.layer.9.attention.output.LayerNorm.bias', 'text_model.encoder.layer.9.attention.output.LayerNorm.weight', 'text_model.encoder.layer.9.attention.output.dense.bias', 'text_model.encoder.layer.9.attention.output.dense.weight', 'text_model.encoder.layer.9.attention.self.key.bias', 'text_model.encoder.layer.9.attention.self.key.weight', 'text_model.encoder.layer.9.attention.self.query.bias', 'text_model.encoder.layer.9.attention.self.query.weight', 'text_model.encoder.layer.9.attention.self.value.bias', 'text_model.encoder.layer.9.attention.self.value.weight', 'text_model.encoder.layer.9.intermediate.dense.bias', 'text_model.encoder.layer.9.intermediate.dense.weight', 'text_model.encoder.layer.9.output.LayerNorm.bias', 'text_model.encoder.layer.9.output.LayerNorm.weight', 'text_model.encoder.layer.9.output.dense.bias', 'text_model.encoder.layer.9.output.dense.weight', 'text_projection.weight', 'vision_model.embeddings.class_embedding', 'vision_model.embeddings.patch_embedding.weight', 'vision_model.embeddings.position_embedding.weight', 'vision_model.embeddings.position_ids', 'vision_model.encoder.layers.0.layer_norm1.bias', 'vision_model.encoder.layers.0.layer_norm1.weight', 'vision_model.encoder.layers.0.layer_norm2.bias', 'vision_model.encoder.layers.0.layer_norm2.weight', 'vision_model.encoder.layers.0.mlp.fc1.bias', 'vision_model.encoder.layers.0.mlp.fc1.weight', 'vision_model.encoder.layers.0.mlp.fc2.bias', 'vision_model.encoder.layers.0.mlp.fc2.weight', 'vision_model.encoder.layers.0.self_attn.k_proj.bias', 'vision_model.encoder.layers.0.self_attn.k_proj.weight', 'vision_model.encoder.layers.0.self_attn.out_proj.bias', 'vision_model.encoder.layers.0.self_attn.out_proj.weight', 'vision_model.encoder.layers.0.self_attn.q_proj.bias', 'vision_model.encoder.layers.0.self_attn.q_proj.weight', 'vision_model.encoder.layers.0.self_attn.v_proj.bias', 'vision_model.encoder.layers.0.self_attn.v_proj.weight', 'vision_model.encoder.layers.1.layer_norm1.bias', 'vision_model.encoder.layers.1.layer_norm1.weight', 'vision_model.encoder.layers.1.layer_norm2.bias', 'vision_model.encoder.layers.1.layer_norm2.weight', 'vision_model.encoder.layers.1.mlp.fc1.bias', 'vision_model.encoder.layers.1.mlp.fc1.weight', 'vision_model.encoder.layers.1.mlp.fc2.bias', 'vision_model.encoder.layers.1.mlp.fc2.weight', 'vision_model.encoder.layers.1.self_attn.k_proj.bias', 'vision_model.encoder.layers.1.self_attn.k_proj.weight', 'vision_model.encoder.layers.1.self_attn.out_proj.bias', 'vision_model.encoder.layers.1.self_attn.out_proj.weight', 'vision_model.encoder.layers.1.self_attn.q_proj.bias', 'vision_model.encoder.layers.1.self_attn.q_proj.weight', 'vision_model.encoder.layers.1.self_attn.v_proj.bias', 'vision_model.encoder.layers.1.self_attn.v_proj.weight', 'vision_model.encoder.layers.10.layer_norm1.bias', 'vision_model.encoder.layers.10.layer_norm1.weight', 'vision_model.encoder.layers.10.layer_norm2.bias', 'vision_model.encoder.layers.10.layer_norm2.weight', 'vision_model.encoder.layers.10.mlp.fc1.bias', 'vision_model.encoder.layers.10.mlp.fc1.weight', 'vision_model.encoder.layers.10.mlp.fc2.bias', 'vision_model.encoder.layers.10.mlp.fc2.weight', 'vision_model.encoder.layers.10.self_attn.k_proj.bias', 'vision_model.encoder.layers.10.self_attn.k_proj.weight', 'vision_model.encoder.layers.10.self_attn.out_proj.bias', 'vision_model.encoder.layers.10.self_attn.out_proj.weight', 'vision_model.encoder.layers.10.self_attn.q_proj.bias', 'vision_model.encoder.layers.10.self_attn.q_proj.weight', 'vision_model.encoder.layers.10.self_attn.v_proj.bias', 'vision_model.encoder.layers.10.self_attn.v_proj.weight', 'vision_model.encoder.layers.11.layer_norm1.bias', 'vision_model.encoder.layers.11.layer_norm1.weight', 'vision_model.encoder.layers.11.layer_norm2.bias', 'vision_model.encoder.layers.11.layer_norm2.weight', 'vision_model.encoder.layers.11.mlp.fc1.bias', 'vision_model.encoder.layers.11.mlp.fc1.weight', 'vision_model.encoder.layers.11.mlp.fc2.bias', 'vision_model.encoder.layers.11.mlp.fc2.weight', 'vision_model.encoder.layers.11.self_attn.k_proj.bias', 'vision_model.encoder.layers.11.self_attn.k_proj.weight', 'vision_model.encoder.layers.11.self_attn.out_proj.bias', 'vision_model.encoder.layers.11.self_attn.out_proj.weight', 'vision_model.encoder.layers.11.self_attn.q_proj.bias', 'vision_model.encoder.layers.11.self_attn.q_proj.weight', 'vision_model.encoder.layers.11.self_attn.v_proj.bias', 'vision_model.encoder.layers.11.self_attn.v_proj.weight', 'vision_model.encoder.layers.2.layer_norm1.bias', 'vision_model.encoder.layers.2.layer_norm1.weight', 'vision_model.encoder.layers.2.layer_norm2.bias', 'vision_model.encoder.layers.2.layer_norm2.weight', 'vision_model.encoder.layers.2.mlp.fc1.bias', 'vision_model.encoder.layers.2.mlp.fc1.weight', 'vision_model.encoder.layers.2.mlp.fc2.bias', 'vision_model.encoder.layers.2.mlp.fc2.weight', 'vision_model.encoder.layers.2.self_attn.k_proj.bias', 'vision_model.encoder.layers.2.self_attn.k_proj.weight', 'vision_model.encoder.layers.2.self_attn.out_proj.bias', 'vision_model.encoder.layers.2.self_attn.out_proj.weight', 'vision_model.encoder.layers.2.self_attn.q_proj.bias', 'vision_model.encoder.layers.2.self_attn.q_proj.weight', 'vision_model.encoder.layers.2.self_attn.v_proj.bias', 'vision_model.encoder.layers.2.self_attn.v_proj.weight', 'vision_model.encoder.layers.3.layer_norm1.bias', 'vision_model.encoder.layers.3.layer_norm1.weight', 'vision_model.encoder.layers.3.layer_norm2.bias', 'vision_model.encoder.layers.3.layer_norm2.weight', 'vision_model.encoder.layers.3.mlp.fc1.bias', 'vision_model.encoder.layers.3.mlp.fc1.weight', 'vision_model.encoder.layers.3.mlp.fc2.bias', 'vision_model.encoder.layers.3.mlp.fc2.weight', 'vision_model.encoder.layers.3.self_attn.k_proj.bias', 'vision_model.encoder.layers.3.self_attn.k_proj.weight', 'vision_model.encoder.layers.3.self_attn.out_proj.bias', 'vision_model.encoder.layers.3.self_attn.out_proj.weight', 'vision_model.encoder.layers.3.self_attn.q_proj.bias', 'vision_model.encoder.layers.3.self_attn.q_proj.weight', 'vision_model.encoder.layers.3.self_attn.v_proj.bias', 'vision_model.encoder.layers.3.self_attn.v_proj.weight', 'vision_model.encoder.layers.4.layer_norm1.bias', 'vision_model.encoder.layers.4.layer_norm1.weight', 'vision_model.encoder.layers.4.layer_norm2.bias', 'vision_model.encoder.layers.4.layer_norm2.weight', 'vision_model.encoder.layers.4.mlp.fc1.bias', 'vision_model.encoder.layers.4.mlp.fc1.weight', 'vision_model.encoder.layers.4.mlp.fc2.bias', 'vision_model.encoder.layers.4.mlp.fc2.weight', 'vision_model.encoder.layers.4.self_attn.k_proj.bias', 'vision_model.encoder.layers.4.self_attn.k_proj.weight', 'vision_model.encoder.layers.4.self_attn.out_proj.bias', 'vision_model.encoder.layers.4.self_attn.out_proj.weight', 'vision_model.encoder.layers.4.self_attn.q_proj.bias', 'vision_model.encoder.layers.4.self_attn.q_proj.weight', 'vision_model.encoder.layers.4.self_attn.v_proj.bias', 'vision_model.encoder.layers.4.self_attn.v_proj.weight', 'vision_model.encoder.layers.5.layer_norm1.bias', 'vision_model.encoder.layers.5.layer_norm1.weight', 'vision_model.encoder.layers.5.layer_norm2.bias', 'vision_model.encoder.layers.5.layer_norm2.weight', 'vision_model.encoder.layers.5.mlp.fc1.bias', 'vision_model.encoder.layers.5.mlp.fc1.weight', 'vision_model.encoder.layers.5.mlp.fc2.bias', 'vision_model.encoder.layers.5.mlp.fc2.weight', 'vision_model.encoder.layers.5.self_attn.k_proj.bias', 'vision_model.encoder.layers.5.self_attn.k_proj.weight', 'vision_model.encoder.layers.5.self_attn.out_proj.bias', 'vision_model.encoder.layers.5.self_attn.out_proj.weight', 'vision_model.encoder.layers.5.self_attn.q_proj.bias', 'vision_model.encoder.layers.5.self_attn.q_proj.weight', 'vision_model.encoder.layers.5.self_attn.v_proj.bias', 'vision_model.encoder.layers.5.self_attn.v_proj.weight', 'vision_model.encoder.layers.6.layer_norm1.bias', 'vision_model.encoder.layers.6.layer_norm1.weight', 'vision_model.encoder.layers.6.layer_norm2.bias', 'vision_model.encoder.layers.6.layer_norm2.weight', 'vision_model.encoder.layers.6.mlp.fc1.bias', 'vision_model.encoder.layers.6.mlp.fc1.weight', 'vision_model.encoder.layers.6.mlp.fc2.bias', 'vision_model.encoder.layers.6.mlp.fc2.weight', 'vision_model.encoder.layers.6.self_attn.k_proj.bias', 'vision_model.encoder.layers.6.self_attn.k_proj.weight', 'vision_model.encoder.layers.6.self_attn.out_proj.bias', 'vision_model.encoder.layers.6.self_attn.out_proj.weight', 'vision_model.encoder.layers.6.self_attn.q_proj.bias', 'vision_model.encoder.layers.6.self_attn.q_proj.weight', 'vision_model.encoder.layers.6.self_attn.v_proj.bias', 'vision_model.encoder.layers.6.self_attn.v_proj.weight', 'vision_model.encoder.layers.7.layer_norm1.bias', 'vision_model.encoder.layers.7.layer_norm1.weight', 'vision_model.encoder.layers.7.layer_norm2.bias', 'vision_model.encoder.layers.7.layer_norm2.weight', 'vision_model.encoder.layers.7.mlp.fc1.bias', 'vision_model.encoder.layers.7.mlp.fc1.weight', 'vision_model.encoder.layers.7.mlp.fc2.bias', 'vision_model.encoder.layers.7.mlp.fc2.weight', 'vision_model.encoder.layers.7.self_attn.k_proj.bias', 'vision_model.encoder.layers.7.self_attn.k_proj.weight', 'vision_model.encoder.layers.7.self_attn.out_proj.bias', 'vision_model.encoder.layers.7.self_attn.out_proj.weight', 'vision_model.encoder.layers.7.self_attn.q_proj.bias', 'vision_model.encoder.layers.7.self_attn.q_proj.weight', 'vision_model.encoder.layers.7.self_attn.v_proj.bias', 'vision_model.encoder.layers.7.self_attn.v_proj.weight', 'vision_model.encoder.layers.8.layer_norm1.bias', 'vision_model.encoder.layers.8.layer_norm1.weight', 'vision_model.encoder.layers.8.layer_norm2.bias', 'vision_model.encoder.layers.8.layer_norm2.weight', 'vision_model.encoder.layers.8.mlp.fc1.bias', 'vision_model.encoder.layers.8.mlp.fc1.weight', 'vision_model.encoder.layers.8.mlp.fc2.bias', 'vision_model.encoder.layers.8.mlp.fc2.weight', 'vision_model.encoder.layers.8.self_attn.k_proj.bias', 'vision_model.encoder.layers.8.self_attn.k_proj.weight', 'vision_model.encoder.layers.8.self_attn.out_proj.bias', 'vision_model.encoder.layers.8.self_attn.out_proj.weight', 'vision_model.encoder.layers.8.self_attn.q_proj.bias', 'vision_model.encoder.layers.8.self_attn.q_proj.weight', 'vision_model.encoder.layers.8.self_attn.v_proj.bias', 'vision_model.encoder.layers.8.self_attn.v_proj.weight', 'vision_model.encoder.layers.9.layer_norm1.bias', 'vision_model.encoder.layers.9.layer_norm1.weight', 'vision_model.encoder.layers.9.layer_norm2.bias', 'vision_model.encoder.layers.9.layer_norm2.weight', 'vision_model.encoder.layers.9.mlp.fc1.bias', 'vision_model.encoder.layers.9.mlp.fc1.weight', 'vision_model.encoder.layers.9.mlp.fc2.bias', 'vision_model.encoder.layers.9.mlp.fc2.weight', 'vision_model.encoder.layers.9.self_attn.k_proj.bias', 'vision_model.encoder.layers.9.self_attn.k_proj.weight', 'vision_model.encoder.layers.9.self_attn.out_proj.bias', 'vision_model.encoder.layers.9.self_attn.out_proj.weight', 'vision_model.encoder.layers.9.self_attn.q_proj.bias', 'vision_model.encoder.layers.9.self_attn.q_proj.weight', 'vision_model.encoder.layers.9.self_attn.v_proj.bias', 'vision_model.encoder.layers.9.self_attn.v_proj.weight', 'vision_model.post_layernorm.bias', 'vision_model.post_layernorm.weight', 'vision_model.pre_layrnorm.bias', 'vision_model.pre_layrnorm.weight', 'visual_projection.weight']
- This IS expected if you are initializing ChineseCLIPTextModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ChineseCLIPTextModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of ChineseCLIPTextModel were not initialized from the model checkpoint at OFA-Sys/chinese-clip-vit-base-patch16 and are newly initialized: ['embeddings.LayerNorm.bias', 'embeddings.LayerNorm.weight', 'embeddings.position_embeddings.weight', 'embeddings.token_type_embeddings.weight', 'embeddings.word_embeddings.weight', 'encoder.layer.0.attention.output.LayerNorm.bias', 'encoder.layer.0.attention.output.LayerNorm.weight', 'encoder.layer.0.attention.output.dense.bias', 'encoder.layer.0.attention.output.dense.weight', 'encoder.layer.0.attention.self.key.bias', 'encoder.layer.0.attention.self.key.weight', 'encoder.layer.0.attention.self.query.bias', 'encoder.layer.0.attention.self.query.weight', 'encoder.layer.0.attention.self.value.bias', 'encoder.layer.0.attention.self.value.weight', 'encoder.layer.0.intermediate.dense.bias', 'encoder.layer.0.intermediate.dense.weight', 'encoder.layer.0.output.LayerNorm.bias', 'encoder.layer.0.output.LayerNorm.weight', 'encoder.layer.0.output.dense.bias', 'encoder.layer.0.output.dense.weight', 'encoder.layer.1.attention.output.LayerNorm.bias', 'encoder.layer.1.attention.output.LayerNorm.weight', 'encoder.layer.1.attention.output.dense.bias', 'encoder.layer.1.attention.output.dense.weight', 'encoder.layer.1.attention.self.key.bias', 'encoder.layer.1.attention.self.key.weight', 'encoder.layer.1.attention.self.query.bias', 'encoder.layer.1.attention.self.query.weight', 'encoder.layer.1.attention.self.value.bias', 'encoder.layer.1.attention.self.value.weight', 'encoder.layer.1.intermediate.dense.bias', 'encoder.layer.1.intermediate.dense.weight', 'encoder.layer.1.output.LayerNorm.bias', 'encoder.layer.1.output.LayerNorm.weight', 'encoder.layer.1.output.dense.bias', 'encoder.layer.1.output.dense.weight', 'encoder.layer.10.attention.output.LayerNorm.bias', 'encoder.layer.10.attention.output.LayerNorm.weight', 'encoder.layer.10.attention.output.dense.bias', 'encoder.layer.10.attention.output.dense.weight', 'encoder.layer.10.attention.self.key.bias', 'encoder.layer.10.attention.self.key.weight', 'encoder.layer.10.attention.self.query.bias', 'encoder.layer.10.attention.self.query.weight', 'encoder.layer.10.attention.self.value.bias', 'encoder.layer.10.attention.self.value.weight', 'encoder.layer.10.intermediate.dense.bias', 'encoder.layer.10.intermediate.dense.weight', 'encoder.layer.10.output.LayerNorm.bias', 'encoder.layer.10.output.LayerNorm.weight', 'encoder.layer.10.output.dense.bias', 'encoder.layer.10.output.dense.weight', 'encoder.layer.11.attention.output.LayerNorm.bias', 'encoder.layer.11.attention.output.LayerNorm.weight', 'encoder.layer.11.attention.output.dense.bias', 'encoder.layer.11.attention.output.dense.weight', 'encoder.layer.11.attention.self.key.bias', 'encoder.layer.11.attention.self.key.weight', 'encoder.layer.11.attention.self.query.bias', 'encoder.layer.11.attention.self.query.weight', 'encoder.layer.11.attention.self.value.bias', 'encoder.layer.11.attention.self.value.weight', 'encoder.layer.11.intermediate.dense.bias', 'encoder.layer.11.intermediate.dense.weight', 'encoder.layer.11.output.LayerNorm.bias', 'encoder.layer.11.output.LayerNorm.weight', 'encoder.layer.11.output.dense.bias', 'encoder.layer.11.output.dense.weight', 'encoder.layer.2.attention.output.LayerNorm.bias', 'encoder.layer.2.attention.output.LayerNorm.weight', 'encoder.layer.2.attention.output.dense.bias', 'encoder.layer.2.attention.output.dense.weight', 'encoder.layer.2.attention.self.key.bias', 'encoder.layer.2.attention.self.key.weight', 'encoder.layer.2.attention.self.query.bias', 'encoder.layer.2.attention.self.query.weight', 'encoder.layer.2.attention.self.value.bias', 'encoder.layer.2.attention.self.value.weight', 'encoder.layer.2.intermediate.dense.bias', 'encoder.layer.2.intermediate.dense.weight', 'encoder.layer.2.output.LayerNorm.bias', 'encoder.layer.2.output.LayerNorm.weight', 'encoder.layer.2.output.dense.bias', 'encoder.layer.2.output.dense.weight', 'encoder.layer.3.attention.output.LayerNorm.bias', 'encoder.layer.3.attention.output.LayerNorm.weight', 'encoder.layer.3.attention.output.dense.bias', 'encoder.layer.3.attention.output.dense.weight', 'encoder.layer.3.attention.self.key.bias', 'encoder.layer.3.attention.self.key.weight', 'encoder.layer.3.attention.self.query.bias', 'encoder.layer.3.attention.self.query.weight', 'encoder.layer.3.attention.self.value.bias', 'encoder.layer.3.attention.self.value.weight', 'encoder.layer.3.intermediate.dense.bias', 'encoder.layer.3.intermediate.dense.weight', 'encoder.layer.3.output.LayerNorm.bias', 'encoder.layer.3.output.LayerNorm.weight', 'encoder.layer.3.output.dense.bias', 'encoder.layer.3.output.dense.weight', 'encoder.layer.4.attention.output.LayerNorm.bias', 'encoder.layer.4.attention.output.LayerNorm.weight', 'encoder.layer.4.attention.output.dense.bias', 'encoder.layer.4.attention.output.dense.weight', 'encoder.layer.4.attention.self.key.bias', 'encoder.layer.4.attention.self.key.weight', 'encoder.layer.4.attention.self.query.bias', 'encoder.layer.4.attention.self.query.weight', 'encoder.layer.4.attention.self.value.bias', 'encoder.layer.4.attention.self.value.weight', 'encoder.layer.4.intermediate.dense.bias', 'encoder.layer.4.intermediate.dense.weight', 'encoder.layer.4.output.LayerNorm.bias', 'encoder.layer.4.output.LayerNorm.weight', 'encoder.layer.4.output.dense.bias', 'encoder.layer.4.output.dense.weight', 'encoder.layer.5.attention.output.LayerNorm.bias', 'encoder.layer.5.attention.output.LayerNorm.weight', 'encoder.layer.5.attention.output.dense.bias', 'encoder.layer.5.attention.output.dense.weight', 'encoder.layer.5.attention.self.key.bias', 'encoder.layer.5.attention.self.key.weight', 'encoder.layer.5.attention.self.query.bias', 'encoder.layer.5.attention.self.query.weight', 'encoder.layer.5.attention.self.value.bias', 'encoder.layer.5.attention.self.value.weight', 'encoder.layer.5.intermediate.dense.bias', 'encoder.layer.5.intermediate.dense.weight', 'encoder.layer.5.output.LayerNorm.bias', 'encoder.layer.5.output.LayerNorm.weight', 'encoder.layer.5.output.dense.bias', 'encoder.layer.5.output.dense.weight', 'encoder.layer.6.attention.output.LayerNorm.bias', 'encoder.layer.6.attention.output.LayerNorm.weight', 'encoder.layer.6.attention.output.dense.bias', 'encoder.layer.6.attention.output.dense.weight', 'encoder.layer.6.attention.self.key.bias', 'encoder.layer.6.attention.self.key.weight', 'encoder.layer.6.attention.self.query.bias', 'encoder.layer.6.attention.self.query.weight', 'encoder.layer.6.attention.self.value.bias', 'encoder.layer.6.attention.self.value.weight', 'encoder.layer.6.intermediate.dense.bias', 'encoder.layer.6.intermediate.dense.weight', 'encoder.layer.6.output.LayerNorm.bias', 'encoder.layer.6.output.LayerNorm.weight', 'encoder.layer.6.output.dense.bias', 'encoder.layer.6.output.dense.weight', 'encoder.layer.7.attention.output.LayerNorm.bias', 'encoder.layer.7.attention.output.LayerNorm.weight', 'encoder.layer.7.attention.output.dense.bias', 'encoder.layer.7.attention.output.dense.weight', 'encoder.layer.7.attention.self.key.bias', 'encoder.layer.7.attention.self.key.weight', 'encoder.layer.7.attention.self.query.bias', 'encoder.layer.7.attention.self.query.weight', 'encoder.layer.7.attention.self.value.bias', 'encoder.layer.7.attention.self.value.weight', 'encoder.layer.7.intermediate.dense.bias', 'encoder.layer.7.intermediate.dense.weight', 'encoder.layer.7.output.LayerNorm.bias', 'encoder.layer.7.output.LayerNorm.weight', 'encoder.layer.7.output.dense.bias', 'encoder.layer.7.output.dense.weight', 'encoder.layer.8.attention.output.LayerNorm.bias', 'encoder.layer.8.attention.output.LayerNorm.weight', 'encoder.layer.8.attention.output.dense.bias', 'encoder.layer.8.attention.output.dense.weight', 'encoder.layer.8.attention.self.key.bias', 'encoder.layer.8.attention.self.key.weight', 'encoder.layer.8.attention.self.query.bias', 'encoder.layer.8.attention.self.query.weight', 'encoder.layer.8.attention.self.value.bias', 'encoder.layer.8.attention.self.value.weight', 'encoder.layer.8.intermediate.dense.bias', 'encoder.layer.8.intermediate.dense.weight', 'encoder.layer.8.output.LayerNorm.bias', 'encoder.layer.8.output.LayerNorm.weight', 'encoder.layer.8.output.dense.bias', 'encoder.layer.8.output.dense.weight', 'encoder.layer.9.attention.output.LayerNorm.bias', 'encoder.layer.9.attention.output.LayerNorm.weight', 'encoder.layer.9.attention.output.dense.bias', 'encoder.layer.9.attention.output.dense.weight', 'encoder.layer.9.attention.self.key.bias', 'encoder.layer.9.attention.self.key.weight', 'encoder.layer.9.attention.self.query.bias', 'encoder.layer.9.attention.self.query.weight', 'encoder.layer.9.attention.self.value.bias', 'encoder.layer.9.attention.self.value.weight', 'encoder.layer.9.intermediate.dense.bias', 'encoder.layer.9.intermediate.dense.weight', 'encoder.layer.9.output.LayerNorm.bias', 'encoder.layer.9.output.LayerNorm.weight', 'encoder.layer.9.output.dense.bias', 'encoder.layer.9.output.dense.weight', 'pooler.dense.bias', 'pooler.dense.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import AutoTokenizer, ChineseCLIPTextModel
import torch

tokenizer = AutoTokenizer.from_pretrained("OFA-Sys/chinese-clip-vit-base-patch16")
model = ChineseCLIPTextModel.from_pretrained("OFA-Sys/chinese-clip-vit-base-patch16")

inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)

last_hidden_states = outputs.last_hidden_state

### Expected behavior

Hopefully, this will make the parameters of the model match the parameters of the pre-trained weights.' text_model.'

Comments:
Hey @ranck626, do you have this error if instead of instantiating `ChineseCLIPTextModel`, you instantiate `ChineseCLIPModel` ?

@LysandreJik 
Only error when loading ChineseCLIPTextModel, ChineseCLIPModel can be loaded and used normally, and even ChineseCLIPVisionModel can be used normally.

This is a bug in the code logic that causes ChineseTextModel to not load the pre-training parameters correctly.



Understood, thanks! @qubvel or @molbap, would you mind taking a look at this if you have bandwidth?

Hi! I think it's just that `model.text_model` is defined as `ChineseCLIPTextModel` in here, so the keys are not matching, there's an extra prefix. 
There's 2 reasons for that: 
1)  because `ChineseCLIPTextModel` is directly defined as the model, instead of for instance
```python
class ChineseCLIPTextModel(PreTrainedModel):
    def __init__(self, config):
        self.text_model = ChineseCLIPTextTransformer(config)
```

I made a workaround in the linked PR: https://github.com/huggingface/transformers/pull/32291 , do you mind trying it out and let me know? Now, this will output a `logger.warning` instead of a lower `logger.info` and that's because of the 2) reason:  the `architectures` key in the `config.json` is explicitly named `ChineseCLIPTextModel`, which will raise the logger level from INFO to WARNING, see here: https://github.com/huggingface/transformers/blob/3fbaaaa64d1ef3d8327adb577994d3d11277c77a/src/transformers/modeling_utils.py#L4461 
Not certain how to change that just yet. 
 


@molbap Thank you for your reply.
You have raised the results of this error to warning level, but there is currently no concrete way to resolve this issue. Is this what you mean?

I tried importing the model and weights using the from pretrained method. At this time, the weight import failed, and then I manually deleted the extra prefix and re-imported the weights. Is it okay to think this way?



from PIL import Image
import requests
from transformers import AutoProcessor, ChineseCLIPVisionModel, ChineseCLIPModel
import torch.nn as nn
import torch
import torch.nn.functional as F

import torch
from transformers import ChineseCLIPTextConfig, ChineseCLIPTextModel, AutoProcessor
clip = ChineseCLIPModel.from_pretrained("OFA-Sys/chinese-clip-vit-base-patch16")
model = ChineseCLIPTextModel.from_pretrained("OFA-Sys/chinese-clip-vit-base-patch16")
processor = AutoProcessor.from_pretrained("OFA-Sys/chinese-clip-vit-base-patch16")
weights_path = "/data/models/OFA-Sys/chinese-clip-vit-base-patch16/clip_cn_vit-b-16.pt" 

# Loading the weights from the .pt file
checkpoint = torch.load(weights_path)

# Extract the actual state_dict
state_dict = checkpoint.get('state_dict', checkpoint)

# Create an updated state_dict by removing the 'module.bert' prefix
updated_state_dict = {
    k.replace('module.bert.', ''): v for k, v in state_dict.items() if k.startswith('module.bert.')
}

# Check if pooler.dense parameters are in state_dict, if not add them
if 'pooler.dense.weight' not in updated_state_dict:
    updated_state_dict['pooler.dense.weight'] = model.pooler.dense.weight
if 'pooler.dense.bias' not in updated_state_dict:
    updated_state_dict['pooler.dense.bias'] = model.pooler.dense.bias

# Handle the case where the embeddings size do not match
if model.embeddings.word_embeddings.weight.size() != updated_state_dict['embeddings.word_embeddings.weight'].size():
    print("fuck you")
    embedding_size = updated_state_dict['embeddings.word_embeddings.weight'].size(0)
    model.embeddings.word_embeddings = torch.nn.Embedding(embedding_size, model.config.hidden_size)
    print(f"Resized model embeddings to match pre-trained embeddings size: {embedding_size}")

# Load the updated state_dict into the model
model.load_state_dict(updated_state_dict)

class ChineseCLIPTextModelWithProjection(nn.Module): # ChineseCLIPVisionWithProjection
    def __init__(self, ChineseCLIPVision):
        super(ChineseCLIPTextModelWithProjection, self).__init__()
        self.model = model
        self.text_projection = nn.Linear(768, 512, bias = False)

    def forward(self, input_ids, attention_mask=None, token_type_ids=None):
        outputs = self.model(input_ids=input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids)
        pooled_output = outputs[0][:, 0, :]  
        text_features = self.text_projection(pooled_output)  
        return text_features

TextModel = ChineseCLIPTextModelWithProjection(model)
new_linear_weight = torch.load('/data/codes/MRAR/projection_weight/text_projection_weight.pt')
TextModel.text_projection.weight.data.copy_(new_linear_weight)

inputA = processor(text="友谊地久天长", return_tensors="pt", padding=True)
inputB = processor(text="友谊地久天长", return_tensors="pt", padding=True)

t1 = TextModel(**inputA)
t1 = t1 / t1.norm(p=2, dim=-1, keepdim=True)
t2 = clip.get_text_features(**inputB)
t2 = t2 / t2.norm(p=2, dim=-1, keepdim=True)

similarity = F.cosine_similarity(t1, t2).item()
print(f"Cosine similarity: {similarity}")

The way you do it seems fine except the word embeddings, if the sizes do not match, doing it the way you are will reset the word embeddings entirely and yield poor results. But yes, it is absolutely fine to change the prefix and load the state_dict manually. Regarding the error level, we can't suppress entirely the message anyway: the checkpoint has text model + image model weights. You're loading text model weights, hence image model weights are unexpected, and it needs to be handled in some way, that way is passing it to `logger.info` if it's a "normal" case. 


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.