TransformerTemporalModel
A Transformer model for video-like data.
TransformerTemporalModel
class diffusers.models.TransformerTemporalModel
< source >( num_attention_heads: int = 16 attention_head_dim: int = 88 in_channels: Optional = None out_channels: Optional = None num_layers: int = 1 dropout: float = 0.0 norm_num_groups: int = 32 cross_attention_dim: Optional = None attention_bias: bool = False sample_size: Optional = None activation_fn: str = 'geglu' norm_elementwise_affine: bool = True double_self_attention: bool = True positional_embeddings: Optional = None num_positional_embeddings: Optional = None )
Parameters
- num_attention_heads (
int
, optional, defaults to 16) — The number of heads to use for multi-head attention. - attention_head_dim (
int
, optional, defaults to 88) — The number of channels in each head. - in_channels (
int
, optional) — The number of channels in the input and output (specify if the input is continuous). - num_layers (
int
, optional, defaults to 1) — The number of layers of Transformer blocks to use. - dropout (
float
, optional, defaults to 0.0) — The dropout probability to use. - cross_attention_dim (
int
, optional) — The number ofencoder_hidden_states
dimensions to use. - attention_bias (
bool
, optional) — Configure if theTransformerBlock
attention should contain a bias parameter. - sample_size (
int
, optional) — The width of the latent images (specify if the input is discrete). This is fixed during training since it is used to learn a number of position embeddings. - activation_fn (
str
, optional, defaults to"geglu"
) — Activation function to use in feed-forward. Seediffusers.models.activations.get_activation
for supported activation functions. - norm_elementwise_affine (
bool
, optional) — Configure if theTransformerBlock
should use learnable elementwise affine parameters for normalization. - double_self_attention (
bool
, optional) — Configure if eachTransformerBlock
should contain two self-attention layers. positional_embeddings — (str
, optional): The type of positional embeddings to apply to the sequence input before passing use. num_positional_embeddings — (int
, optional): The maximum length of the sequence over which to apply positional embeddings.
A Transformer model for video-like data.
forward
< source >( hidden_states: Tensor encoder_hidden_states: Optional = None timestep: Optional = None class_labels: LongTensor = None num_frames: int = 1 cross_attention_kwargs: Optional = None return_dict: bool = True ) → TransformerTemporalModelOutput or tuple
Parameters
- hidden_states (
torch.LongTensor
of shape(batch size, num latent pixels)
if discrete,torch.Tensor
of shape(batch size, channel, height, width)
if continuous) — Input hidden_states. - encoder_hidden_states (
torch.LongTensor
of shape(batch size, encoder_hidden_states dim)
, optional) — Conditional embeddings for cross attention layer. If not given, cross-attention defaults to self-attention. - timestep (
torch.LongTensor
, optional) — Used to indicate denoising step. Optional timestep to be applied as an embedding inAdaLayerNorm
. - class_labels (
torch.LongTensor
of shape(batch size, num classes)
, optional) — Used to indicate class labels conditioning. Optional class labels to be applied as an embedding inAdaLayerZeroNorm
. - num_frames (
int
, optional, defaults to 1) — The number of frames to be processed per batch. This is used to reshape the hidden states. - cross_attention_kwargs (
dict
, optional) — A kwargs dictionary that if specified is passed along to theAttentionProcessor
as defined underself.processor
in diffusers.models.attention_processor. - return_dict (
bool
, optional, defaults toTrue
) — Whether or not to return a TransformerTemporalModelOutput instead of a plain tuple.
Returns
TransformerTemporalModelOutput or tuple
If return_dict
is True, an
TransformerTemporalModelOutput is returned, otherwise a
tuple
where the first element is the sample tensor.
The TransformerTemporal
forward method.
TransformerTemporalModelOutput
class diffusers.models.transformers.transformer_temporal.TransformerTemporalModelOutput
< source >( sample: Tensor )
The output of TransformerTemporalModel
.