A method comprising: dividing a video into a plurality of video shots; analyzing each frame in a video shot from the plurality of video shots; determining a visual saliency and an audio saliency of each frame of the video shot; determining a key frame of the video shot based on the saliency of each frame of the video shot; extracting visual features from the key frame; extracting Bag-of-Word descriptors from the extracted visual features, and cluster the Bag-of-Word descriptors to concept patterns by spectral clustering; performing shot clustering of the plurality of video shots based on the concept patterns; using a saliency tuning method comprising scaling the visual saliency value of a respective frame by a corresponding camera attention factor determined by camera motion; and fusing the visual saliency and the audio salience for each frame into a fused saliency; and determining the saliency X(i) for shot i as the average fused saliency of the frames in shot i; generating a summary of the video based upon a constrained optimization problem, comprising concatenating at least one shot from every concept pattern by solving a maximization problem with a maximization function f(i, j, t), which is defined as f i j t = { max i ≤ k < j t = t 0 + len j / γ f i k t 0 + X j + λ exp − | d j k − d ̅ / β j > i X i j = i − ∞ j < i where f(i, j, t) represents a shot sequence that starts at shot i, ends at shot j and contains γ · t frames, and where len(j) represent the number of frames in shot j, d(j,k) represents the number of frames between shot j and shot k, d represents the average number of frames between two nearby shots in the uniform sampling of shot pairs if all shots are assumed to take the average shot length. dividing a video into a plurality of video shots; analyzing each frame in a video shot from the plurality of video shots; determining a visual saliency and an audio saliency of each frame of the video shot; determining a key frame of the video shot based on the saliency of each frame of the video shot; extracting visual features from the key frame; extracting Bag-of-Word descriptors from the extracted visual features, and cluster the Bag-of-Word descriptors to concept patterns by spectral clustering; performing shot clustering of the plurality of video shots based on the concept patterns; using a saliency tuning method comprising scaling the visual saliency value of a respective frame by a corresponding camera attention factor determined by camera motion; and fusing the visual saliency and the audio salience for each frame into a fused saliency; and determining the saliency X(i) for shot i as the average fused saliency of the frames in shot i; generating a summary of the video based upon a constrained optimization problem, comprising concatenating at least one shot from every concept pattern by solving a maximization problem with a maximization function f(i, j, t), which is defined as f i j t = { max i ≤ k < j t = t 0 + len j / γ f i k t 0 + X j + λ exp − | d j k − d ̅ / β j > i X i j = i − ∞ j < i where f(i, j, t) represents a shot sequence that starts at shot i, ends at shot j and contains γ · t frames, and where len(j) represent the number of frames in shot j, d(j,k) represents the number of frames between shot j and shot k, d represents the average number of frames between two nearby shots in the uniform sampling of shot pairs if all shots are assumed to take the average shot length.