<figure><img src="image_1.png" /></figure>

HAL
open science

# Quality and recognizability estimation video encryption database

Heinz Hofbauer, Florent Autrusseau, Andreas Uhl

► To cite this version:

Heinz Hofbauer, Florent Autrusseau, Andreas Uhl. Quality and recognizability estimation video encryption database. Information Sciences, 2024, 678, pp.120919. 10.1016/j.ins.2024.120919. hal-04814562

HAL Id: hal-04814562

https://nantes-universite.hal.science/hal-04814562v1

Submitted on 2 Dec 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

ELSEVIER

Contents lists available at ScienceDirect

Information Sciences

journal homepage: www.elsevier.com/locate/ins

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

# Quality and recognizability estimation video encryption database

Heinz Hofbauer<sup>a,*</sup>, Florent Autrusseau<sup>b</sup>, Andreas Uhl<sup>a</sup>

## ARTICLE INFO

**Keywords:**

Selective encryption

Visual recognition database

Evaluation of visual quality indices

## ABSTRACT

In literature about selective encryption of image and video content, image quality indices are usually used to gauge the degree of encryption. These methods have frequently been shown not to work well for the evaluation of encryption, mainly due to them being trained on predominantly high quality contents. The problem for creating a proper recognition index or visual encryption strength index is the lack of data to train on. In this paper we present the first database of encrypted video content, ranging from high quality to completely unrecognizable, together with human observer scores for quality and recognizability. We also provide a basic evaluation of visual quality indices on this database, directly and in different combination by fusion, to showcase that currently image and video quality indices are ill fit for the purpose of estimating video encryption strength/recognizably. This inability of quality indices to perform also showcases that this database fills the required blind spot of currently available data.

## 1. Introduction

Selective video encryption is a topic still under active research [30,38,9,21,5,28], as is the related question of creating a visual security index [46,43,45,48]. Due to the nature of selective encryption, that a format compliant decoder should still be able to decode the encrypted bitstream, an evaluation of residual visual information is required. This usually takes the form of using a visual quality index (VQI) to assess the quality, and therefore the residual information, left in the visual parts of the decoded video sequence. If confidentiality is desired, a test for the unrecognizability of the content is performed, again by VQIs as there are no recognizability measures. Typically, VQIs are supposed to assess high quality image/video material, for assessing encoding parameters rather than encryption parameters, and are designed as such. It is no great surprise that VQIs are not very good at dealing with the strong visual interference/distortion resulting from encryption [11,14]. However, there are no alternatives, and works from literature thus use VQIs for security assessment, as an example see the above cited papers which all use SSIM ([38,9,5]) and/or PSNR ([30,38,21,5,28]) the shortcomings of these VQIs as analyzed in [14].

Let us briefly elaborate on the possible goals of selective encryption to make clearer the need for both a quality as well as a recognizability evaluation. Typical goals of selective encryption are: a) Content confidentiality, where the content of an encrypted video should be unintelligible, i.e. unrecognizable; b) Sufficient encryption, where the target is a reduction of quality to such a degree that it is no longer enjoyable to watch, due to high distortions; and c) Transparent encryption, where typically a quality band is given as a target such that the video can be consumed albeit in a reduced quality, as a sort of preview. Clearly for (a) a low quality is not

<footnote><sup>a</sup> Paris Lodron University Salzburg, Department of Artificial Intelligence and Human Interfaces, Jakob Harringer Straße 2, 5020, Salzburg, Austria</footnote>

<footnote><sup>b</sup> TeN UMR-6607, Polytech'Nantes, and Inserm UMR-1087, Institut du Thorax, University of Nantes, 8 quai Moncousu, 44007, Nantes, France</footnote>

<footnote>* Corresponding author.</footnote>

<footnote>E-mail addresses: hofbauer@cs.sbg.ac.at (H. Hofbauer), Florent.Autrusseau@univ-nantes.fr (F. Autrusseau), uhl@cs.sbg.ac.at (A. Uhl).</footnote>
sufficient, we need some form of recognizability measure. For (b) and (c) quality is of importance, although typically outside of the
high quality range which is the typical use-case for encoding optimization.

For the development of visual security indices, i.e., a subset of VQIs that specifically target the low quality output of format
compliant selectively encrypted image or video data, it is important to have a ground-truth to use for evaluation. The question of
evaluating recognition and quality is not an easy one, and it is not made easier by the relative lack of data. Only recently two
databases have been created for still images, [11,8,10], but to the best of the authors knowledge there is still not a single database
for video encryption with accompanying quality and recognition scores based on human observers. While there are attempts, [19],
to transfer data from still image data to video applications the spatio-temporal relation can not be properly captured.

To facilitate research in the area of video encryption evaluation we introduce a, publicly available, database of encrypted video
material with a subjective assessment. The subjective assessment covers not only quality but recognizability of content as well. In
this study, we do not attempt to develop a new VQI, but we will evaluate existing VQIs on the produced data. This will, on one hand,
show that the data produced fills the missing spot for the development of VQIs. On the other hand, the evaluation can inform users
which of the VQIs is the best option for the evaluation of selectively encrypted video sequences for the time being.

It should be noted that, as the name suggests, VQIs are built to estimate quality, not recognition. Hence, as a very first step
towards the understanding of the recognition mechanisms, it is crucial to investigate how quality might (or might not) relate to
recognizability. In this study, we have conducted two separate subjective experiments, one was devoted to collect quality scores, the
other one intended to evaluate the recognizability of the contents.

To summarize our contributions, we: a) create a first database of encrypted videos, spanning quality and recognition scores, collecting human observer scores for both recognition and quality; b) evaluate current, and typically used VQIs, on the database to determine how suitable they are, for their very purpose; and c) investigate the relation of recognition and quality to gain an insight whether or nor a single VQI can be used for both.

The rest of the paper is structured as follows, Section 2 will describe the database, which sequences it contains, how it was encrypted, how the human observer scores were obtained and how the final scores were derived after outlier pruning. Section 3 details our analysis of the database and experiments, an evaluation of current VQIs on the data, and an attempt to use fusion to improve the scores in a straightforward manner. In addition we looked at constituents of motion pictures in separation (they are frequently used in some form in VQIs as well) to see how they behave in encrypted motion pictures. Finally, Section 4 concludes the paper and succinctly summarizes the important results from prior sections.

## 2. Encrypted video database

**Reproducible Research:** The database will be made available at the webpage of the WAVELAB group.<fnref n="1" /> It contains the reference videos as well as the encrypted videos in a YUV format, it also contains the results of the subjective quality and recognition scores, once as obtained, i.e. the unpruned version, and once with outliers removed for direct use.

### 2.1. Database description

The database contains 9 video sequences encrypted with two methods (based on H.265 and motion JPEG 2000) to obtain 180 video sequences ranging from high quality to unrecognizable.

The sequences were obtained from the Video Test Media page at Xiph.org.<fnref n="2" /> The first 150 frames of each sequence were used to get a video of roughly 5 seconds in length with a framerate of 29.97 frames per second. The length of the sequences was chosen in tandem with the number of encryption parameters and the overall number of sequences to be able to acquire the evaluation of either quality or recognition scores in one session without provoking viewer fatigue. The resulting videos were resized to a size of 898 × 504 to be able to present two videos with a small border on a standard 1080p monitor without scaling, a requirement for the quality acquisition protocol. This resulted in a base video to be encrypted. The sequences themselves were selected based on a number of criteria, foremost, that there is no scene change in the selected part of the sequence. A scene change would allow the identification of the original, even if encrypted, due to the clear cut if viewed side by side. Other than that we tried to have each sequence be unique in that a simple set of words could clearly identify it, more on that in discussion of the protocol for recognition acquisition. The next selection criteria was that the video sequences should be comprised of different content, motion types, camera movements and so on, see Table 1. Samples of the sequences included and samples of encrypted frames are presented in Fig. 1.

Apart from the 9 sequences used in the actual database, a 10th sequences, controlledburn (also from Xiph.org), was used for
anchoring the observers. The sequences were of course encrypted with the same parameters as the other sequences.

### 2.2. Encryption of video content

**A note about selective encryption:** The encryption types we use are designed to enable the selective encryption of certain coding
elements with state of the art ciphers. The actual cipher can be exchanged if flaws are found. In this work, we used the AES cipher.
For our case the actual strength of the encryption is also less important as we deal with the visual quality so a replacement of the

<footnote n="1">¹ https://wavelab.at/sources/USEE Vid.</footnote>

<footnote n="2">² https://media.xiph.org/video/derf.</footnote>
<table><caption>**Table 1**
Sequences and their properties as well as the description used in the acquisition of recognition scores.</caption><thead><tr><td>Sequences</td><td>Motion</td><td>Camera</td><td>Description</td></tr></thead><tbody><tr><td>bluesky</td><td>global</td><td>rotation</td><td>Sky and trees</td></tr><tr><td>crowdrun</td><td>local</td><td>slight shake</td><td>Marathon runners</td></tr><tr><td>mobcal</td><td>global</td><td>pan</td><td>Photo of a boat</td></tr><tr><td>oldtowncross</td><td>global</td><td>dolly, slight shake</td><td>Town</td></tr><tr><td>station2</td><td>global</td><td>zoom</td><td>Train station</td></tr><tr><td>fourpeople</td><td>local</td><td>locked</td><td>Office meeting</td></tr><tr><td>pedestrianarea</td><td>local</td><td>slight shake, pan</td><td>Shopping Street</td></tr><tr><td>touchdownpass</td><td>local</td><td>slight shake</td><td>American Football</td></tr><tr><td>vidyo1</td><td>local</td><td>locked</td><td>Office meeting</td></tr></tbody></table>

selected elements with a random sequence should be equally sufficient and represent regular ciphers well, as their output should appear random.

Two methods for encryption were used, one based on Motion JPEG2000 [7,15], as used in digital cinema, and one based on H.265 [36,16,17,32,6], a modern video codec. A couple of things should be noted here. First, the actual selective encryption methods are not important, they are merely a means to obtain a number of images which have a well spaced quality and recognizability index. And second, we will only briefly describe the settings used, going into detail with selective encryption methods, especially format compliant ones, quickly requires detailed knowledge about the codecs involved and would far exceed the scope of this paper.

Motion JPEG2000 (MJ2k) encodes each frame as a still image, and as such a still image encryption method was used. The method from [34] encrypts JPEG2000 files, encoded in resolution progression, by encrypting CCPs (codeblock contribution to packet) of code blocks while maintaining signal markers. The ordering of contributing data in the bit stream is arranged such that the base information comes at the beginning, followed by refinement blocks which bring more and more detail into the image. Consequently, the encryption method can encrypt a window in this bit stream with an offset and a size. The window size in our case was fixed to 50% of the bitstream and the offset was varied. The farther offset from the beginning of the bitstream, the more the encryption is shifted from larger to finer details. That is coarse vs. fine structures, this can be seen (with the actual offsets used in the creation of the database) in Fig. 1. Further, the JJ2000 version used in the experiments includes the patches and enhancements to JPEG2000 error concealment provided by [35]. This provides a better decoding of the frames which minimizes artifacts in the output as much as possible.

For the “High Efficiency Video Coding” (H.265) encryption the selection of what to encrypt it is a bit more complicated. Due to the high compression of the codec it was quite difficult to find a proper spread of encryption parameters which produces the desired output in terms of quality and recognition. We use the software implemented and described in [40]. Due to the high compression rate, it is quite difficult to scale the encryption in terms of quality and recognizability. We overall used a combination of four encryption types from the framework: 12ac encrypting the first and second AC coefficient; mvd encrypts the motion vector differences; nz encrypts the position of non-zero coefficients; and co encrypts the coefficient signs. For the cases where very low quality of non-recognizability was required a single pass was not sufficient, so a loop of encryption, decoding, encoding was used. In this way a sufficiently low quality could be reached while maintaining the appearance of encryption artifacts.

## 2.3. Subjective evaluation protocols

In this work, two distinct subjective experiments have been conducted. A quality assessment (Paired Comparison) and a recognizability protocol were set up. Indeed, one of our goals in this study was to compare subjective (and objective) quality and recognizability scores. At first the authors and some associates ran a pre-experiment in order to determine the rough quality/recognizability ranking of the video dataset under consideration. This was necessary to display the video sequences with an increasing recognizability, i.e. first unrecognizable sequences then more recognizable sequences. If we would not have done this then more recognizable sequences could have given clues, large structures, silhouettes motion types and so on, to identify otherwise less recognizable sequences.

The subjective recognizability protocol was set up as follows (see Fig. 2 for examples of video and selection screen): A selectively encrypted video was played at the center of the monitor, surrounded by a uniform gray background. Each video was 5 seconds long. Once the video was played, a screen with 9 voting options was presented. The voting options were “American Football”, “Train station”, “Office meeting”, “Marathon runners”, “Town”, “Shopping street”, “Sky and trees”, “Photo of a boat”, and finally “Don’t know”. The observer had to select one of these options to move onto the next display. An anchoring sequence was performed prior to running the actual test to familiarize observers with the interface and range of potential distortions to expect. Of course a different sequence was used for anchoring (controlledburn). The test was quite tedious, as, at first, the worst quality sequences were presented, and hence the observers had to run the experiment for a little while before the quality was good enough to make out the content. Of course, toward the end, all sequences were fully recognized. The recognizability score is the percentage of correct votes for a given sequence.

On the other hand, the quality protocol was more classical, two video sequences were played side by side on the monitor. The original sequence was always displayed on the left part of the monitor, whereas the distorted sequence was played on the right
<figure><img src="image_6.png" /><figcaption>(a) Samples of the references sequences.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>(b) Encrypted samples for the motion JPEG2000 encryption on the touchdownpass sequence with window encryption, window size 50% offset as given.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(c) Encrypted samples for the motion HEVC encryption on the touchdownpass sequence with the given parameter set.</figcaption></figure>

<caption>Fig. 1. Samples from the database, in each case frame 10 of the sequences are shown.</caption>

part. After the 5 seconds sequences were played, a voting screen was presented, and the observers had to rate the perceived quality and recognizability among 5 categories: "1: Unrecognizable", "2: Match", "3: Very bad quality", "4: Broadcast bad", and finally "5: Broadcast good". The observers were instructed to choose categories "1" or "2" only if they were unable to make out the content. The second category was devoted to the video sequences for which the observers can assure it is the same content being displayed, only because they can match either a shape, a color, or a motion, but the content itself was not recognized. Labels "1" and "2" had to be included to account for the range of encryption levels even though they are not strictly speaking related to quality.
<figure><img src="image_9.png" /><figcaption>Fig. 2. Recognizability protocol, first a video is shown (left) then a corresponding word label has to be selected (right).</figcaption></figure>

The same set of observers were enrolled for both subjective experiments. Overall, forty eight observers took part in this study. Concerning the recognition protocol, despite the observer selection algorithm (dendrogram), launched on the obtained subjective scores no observers were discarded from the test set, i.e. all the collected subjective scores were statistically coherent. As for the quality assessment experiment, the observer selection algorithm exhibited only one incoherent subjective scoring, and the observer was thus discarded from the dataset. The average observers' age was 33.2 ($\sigma$ = 8.8).

As pointed out in the literature [11,12], when recognizability mechanisms are at stake, the viewing conditions have little to no impact on the subjective evaluation. Illumination conditions, screen characterization or even viewing distance play a very marginal role in the output scores. However, during this experiment, the observers' vision was tested with both a Snellen eye chart and Ishihara color plates, the viewing distance was set to six times the video sequences' height, and the observers were comfortably seated in an individual office so that they wouldn't be disturbed during the subjective test.

We performed outlier pruning on the results to remove observer scores from aberrant or non-attentive observers. We did this by clustering opinion scores in a tree like fashion [26] with joining of clusters based on the distance between cluster opinions. The cutoff for the merging of clusters was set to a z-score of $\mu + 3\sigma$ of the observer values. No outliers were removed from the recognition evaluation and a single outlier was removed for the quality evaluation. As noted above in the publicized database both pruned and un-pruned lists of observer scores are included.

## 3. Analysis and experiments

First we will ascertain that we achieved our goal of creating and scoring data that ranges from high to low quality and from totally unrecognizable to recognizable. Then we will evaluate the Performance of various VQIs on the database, as they are and in combination through fusion. We will also analyze how the number of frames used for quality/recognizability assessment influences the results.

In the following experiments we use a number of different visual quality indices ranging from classic (VIF [33], LEG [13], SSIM [41], CPAI [2], MSSIM [42], and PSNR), including VQIs created with encryption in mind (LSS [23], ESS [23], LE [37], NSD [47], and LFBVS [39]) as well as no-reference indices (BLIINDS-II [31], BRISQUE [25]), to modern CNN based (VGG16 [50], AlexNet [50], full reference TOPIQ [4,3]) and VQIs specifically created for video quality (VMAF [20], NTIA (VQM) [29], Dover [44]). For reasons of brevity we can not go into detail about the individual VQIs, we refer the readers to the individual papers, or, if an overview is desired, to surveys about image [49] and video [24] quality.

### 3.1. Analysis of the data

When running a subjective experiment, at first, we must ensure that the subjective scores do span the entire quality/recognizability range. We can observe of Fig. 3 (left sub-figures) that indeed, both the quality and recognizability scores do use the whole subjective scales. However, as can be observed on the right sub-figures, it appears that very little variance in quality occurs when the content is not fully recognized. This totally makes sense, nevertheless, we might thus envision that none of the available quality indices would perform well at estimating the recognizability index.

As expected, the VQIs do perform better when assessing quality rather than when estimating recognizability (a higher level perceptual mechanism). Before considering objective quality assessment, let us first investigate how subjective quality and recognizability relate to each other. We plot in Fig. 4 the Spearman and Pearson correlations between the subjective quality and recognizability scores. We can observe a somewhat significant performances disparity in between video sequences, and between encryption protocols (H.265 and MJ2k). Hence, a VQI that would offer a high accuracy at estimating quality would inevitably fail given a recognizability evaluation task.

Numerous objective quality indices have been tested, we present in Fig. 5 (Pearson correlation only, for Spearman correlation see the next section), the results of fourteen VQIs. Unsurprisingly, although the quality scores may be reasonably well predicted, the recognizability, on the other hand is much more difficult to estimate. Indeed, as observed on Fig. 5, for each VQI, much higher correlations are obtained when quality scores are matched (left figures) rather than against recognizability scores (right figures). It
<figure><img src="image_10.png" /><figcaption>**Fig. 3.** H.265 and MJ2k subjective scores plotted as a function of the selective encryption algorithm, but with increasing recognition index (left figures) and as a quality versus recognizability point cloud.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>**Fig. 4.** Spearman and Pearson correlations between quality and recognizability subjective scores</figcaption></figure>

is also quite interesting to mention that (lower plots) most VQIs perform significantly better within a higher quality range. Here, we have ordered and split the quality and recognizability scores into three different clusters. The quality clusters were such as: $Q \in [1, 2]$ for the lowest quality, $Q \in [2, 3.5]$ for a medium quality range, and finally, $Q \in [3.5, 5]$ for the best quality. Similarly, recognizability clusters were formed, such that $R \in [0, 0.3]$, $R \in [0.3, 0.9]$ and $R \in [0.9, 1]$. The rationale behind this subjective score decomposition was to investigate whether the VQIs would actually perform differently depending on the quality range. In fact, as already mentioned, VQIs are designed to operate within a relatively high quality range. For the vast majority of metrics, their purpose is to estimate the quality loss induced by some high performances multimedia compression methods, and thus, they are tuned to do well near the visibility threshold. Hence, their performances might not be optimal when highly distorted sequences are considered. This is what we show in the two lower panels of Fig. 5, by splitting the dataset into three different subjective ranges: the lower the contents' quality, the worst the metrics' performances.

## 3.2. Performance of visual quality indices on the database

We evaluate the VQIs by comparing them to the observer scores. Since neither the human observer response nor VQI scores are linear a ranked order correlation is used but converting the scores to ranks, i.e. the second higher score would become rank #2.
<figure><img src="image_12.png" /><figcaption>**Fig. 5.** Pearson correlations computed on the Quality (left) or Recognizability (right) scores. The correlations were either computed on the whole dataset (upper figures) or onto three distinct Q/R sub-clusters (lower figure).</figcaption></figure>

We restrict ourselves to the SROC here, tables are large enough as it is without giving multiple correlation types. Let $H$ be the human observer scores, $S$ the VQI scores, and $R: (R) \to (N)$ a function converting scores to ranks. Then the Spearman rank order correlation is defined as $SROC = \frac{\text{cov}(R(H), R(S))}{\sigma(R(H))\sigma(R(S))}$. The results of a Spearman rank order correlation (SROC) of the visual quality indices (VQI) performances with the mean observer quality and recognition scores can be seen in Table 2.

A brief note about the metric scores and the SROC. Mostly the metrics use a normalized output (in [0,1]), with few exceptions (most notably the PSNR). The scores can be quality scores (higher is a better quality) or distortion scores (higher is a lower quality). The resulting SROC is positive if a quality score is compared to the human observer scores and negative if a distortion score is compared to human observer scores. If the distortion score ($S_d(i)$) is converted to a quality score $S_q(I)$, i.e., $S_q(I) = 1 - S_d(I)$ the sign of the SROC flips but the absolute value stays the same. The scores given in the Table are thus absolute scores.

Now let us look at the overall, i.e., for a combination both encoding types MJ2k and H.265, data, as this would represent a general application (that is one not tailored to a specific encoding type). The results range from surprisingly poor (AlexNet) to relatively acceptable (MSSIM). But overall the scores fall far short of what would be expected from a proper quality/recognition index. It is of course not surprising that the general application (both encoding types at the same time) exhibits a lower performance than a specific application (one encoding). The problem is that the different VQIs vary widely in performance when it comes to the different encoding types, see Table 2 sub-tables for MJ2k an H.265 only. An example of this is the AlexNet VQI which highly correlates with observer scores on the H.265 encryption but has a negative correlation with observer scores for MJ2k! And while this explains why the overall performance of AlexNet for MJ2k and H.265 combined is poor, it does not solve the problem of having to know which VQI plays well with each combination of encoding/encryption type.

To reiterate, some VQIs are able to perform the task of recognition and/or quality assessment for some distortion types, but not others. But the task, recognition/quality assessment, is the same, irrespective of the distortions. Therefore, we don't really have general VQIs, only ones which can work on a specific distortion type.

So in practice, if a new encryption type, even if used on a known encoding, is to be evaluated it is unclear which VQI to use. While this is not new information as such, the shortcoming of VQIs as measures for image security has been discussed in detail in [14,11] for example, it bears repeating that an overall good performance, i.e., a generalizing, quality or recognition index for encrypted media types is desired. The lower performance of the recognition rating, as opposed to the quality rating, as hinted at in Fig. 3 is clearly observable, although not very pronounced.

Interesting is also the comparison to the performance of VQI on still images, most recently in [11], where the performance in terms of quality and recognition is much lower, and the gap between quality and recognition performance is also more pronounced. The likely difference maker is the reduced samples in the database and the higher number metrics scores per sample, 150 for a 5 second 30 fps video vs. 1 for a still image. It should be noted that the VQIs use an aggregate (averaging) of still image (per frame) scores. While this is perfectly serviceable for the MJ2k encoding as it is essentially a sequence of still images it neglects temporal
<caption>Table 2</caption>

Absolute Spearman rank order correlation of averaged per-frame visual quality index (VQI) score with quality (Q) or
recognizability (R) mean observer scores. Once for each encoding type separately and for all combined, ordering is by
quality.

<table><thead><tr><th colspan="3">MJ2k+H.265</th><th colspan="3">MJ2k only</th><th colspan="3">H.265 only</th></tr><tr><th>VQI</th><th>Q</th><th>R</th><th>VQI</th><th>Q</th><th>R</th><th>VQI</th><th>Q</th><th>R</th></tr></thead><tbody><tr><td>MSSIM</td><td>0.7956</td><td>0.7316</td><td>MSSIM</td><td>0.9669</td><td>0.8473</td><td>AlexNet</td><td>0.8695</td><td>0.8156</td></tr><tr><td>LSS</td><td>0.7873</td><td>0.6777</td><td>LSS</td><td>0.9409</td><td>0.7738</td><td>VGG16</td><td>0.8392</td><td>0.7604</td></tr><tr><td>PSNR</td><td>0.7573</td><td>0.6767</td><td>PSNR</td><td>0.8969</td><td>0.7457</td><td>ESS</td><td>0.7771</td><td>0.7197</td></tr><tr><td>NTIA</td><td>0.7485</td><td>0.6884</td><td>CPA1</td><td>0.8899</td><td>0.6632</td><td>LFBVS</td><td>0.7459</td><td>0.6760</td></tr><tr><td>LEG</td><td>0.7218</td><td>0.6657</td><td>LEG</td><td>0.8121</td><td>0.6884</td><td>Dover</td><td>0.7155</td><td>0.6617</td></tr><tr><td>TOPIQ</td><td>0.6765</td><td>0.6003</td><td>TOPIQ</td><td>0.8009</td><td>0.6711</td><td>MSSIM</td><td>0.7113</td><td>0.6520</td></tr><tr><td>Dover</td><td>0.6604</td><td>0.5740</td><td>VIF</td><td>0.7995</td><td>0.6946</td><td>LEG</td><td>0.6496</td><td>0.6159</td></tr><tr><td>VMAF</td><td>0.6460</td><td>0.5532</td><td>SSIM</td><td>0.7944</td><td>0.6809</td><td>NSD</td><td>0.6438</td><td>0.5956</td></tr><tr><td>ESS</td><td>0.6293</td><td>0.5670</td><td>VGG16</td><td>0.7688</td><td>0.6302</td><td>LSS</td><td>0.6091</td><td>0.5633</td></tr><tr><td>VGG16</td><td>0.6112</td><td>0.5601</td><td>NTIA</td><td>0.7461</td><td>0.7603</td><td>PSNR</td><td>0.6077</td><td>0.5563</td></tr><tr><td>CPA1</td><td>0.6082</td><td>0.4741</td><td>BRISQUE</td><td>0.7107</td><td>0.5531</td><td>VMAF</td><td>0.5797</td><td>0.5223</td></tr><tr><td>BLIINDS-II</td><td>0.5001</td><td>0.5245</td><td>Dover</td><td>0.6517</td><td>0.4860</td><td>SSIM</td><td>0.5686</td><td>0.5277</td></tr><tr><td>VIF</td><td>0.4943</td><td>0.4437</td><td>VMAF</td><td>0.6339</td><td>0.6412</td><td>NTIA</td><td>0.5600</td><td>0.5113</td></tr><tr><td>LFBVS</td><td>0.4214</td><td>0.3379</td><td>BLIINDS-II</td><td>0.5002</td><td>0.5365</td><td>TOPIQ</td><td>0.5490</td><td>0.4673</td></tr><tr><td>SSIM</td><td>0.2843</td><td>0.2904</td><td>ESS</td><td>0.4730</td><td>0.3861</td><td>BRISQUE</td><td>0.5155</td><td>0.4731</td></tr><tr><td>BRISQUE</td><td>0.1145</td><td>0.0679</td><td>AlexNet</td><td>0.4661</td><td>0.3525</td><td>BLIINDS-II</td><td>0.5001</td><td>0.5156</td></tr><tr><td>AlexNet</td><td>0.0944</td><td>0.1322</td><td>LFBVS</td><td>0.3678</td><td>0.1517</td><td>CPA1</td><td>0.4083</td><td>0.3408</td></tr><tr><td>NSD</td><td>0.0505</td><td>0.0106</td><td>NSD</td><td>0.2433</td><td>0.4035</td><td>VIF</td><td>0.3901</td><td>0.3423</td></tr><tr><td>LE</td><td>0.0386</td><td>0.0937</td><td>LE</td><td>0.0824</td><td>0.1405</td><td>LE</td><td>0.0102</td><td>0.0143</td></tr></tbody></table>

information, or rather changes of due to encryption, in H.265 which includes temporal information during encoding (through motion vectors). This might also be the reason the peak performance of VQIs for H.265 seems to be lower than for MJ2k if viewed in isolation.

What should also be noted is that the video quality indices are in no way better than the averaged still image score of regular VQIs. It is also clear that both the NTIA and VMAF have been built as general purpose tools, performing better on a mixed evaluation set (MJ2k+H.265) than on each individually. DOVER being a more recent method, was trained heavily on publicly available data which is typically encoded with H.264 or H.265 performs significantly better on that modality.

### 3.3. Learning a better VQI frame score combination

VQIs analyze and score still images, this differs for video quality indices (DOVER, NTIA and VMAF) which score the video as a whole. Up to now we have reduced the 150 data points, one per frame, in a traditional way, by averaging, to a single score. Due to different correlation directions, e.g., AlexNet SROC has a different sign on MJ2k and H.265 (although for ranking purposes this was not shown), the performance overall falls flat. There is a potential for improvement if the per-frame-scores are not averaged but combined in a more complex manner. We use a support vector machine (SVM) for regression (SVR), i.e., fitting the VQI scores to the curve of quality/recognizability as given by human observers. We use a 9-fold cross-validation for this task, leaving one sequence out for testing per fold, meaning one sequence is never seen during training.

The results of this experiment, again in terms of Spearman rank order correlation (SROC) are given in Table 3. Overall, the SVR improves almost all scores, ironically it makes the best prior VQI (MSSIM) slightly worse. Clearly, while this is the potential route of improvement, if a VQI is already a good fit, then averaging seems to be quite sufficient as a score aggregate.

Given that using an SVR does not degrade performance but allows a free combination of per-frame scores an obvious extension would be to combine two VQI scores (two scores per-frame) to get a combination outperforming the individual performances. Results are given in Table 4, where the row and column give the scores fused with SVR, the table will be mirrored below and above the prime diagonal. The diagonal is of course not a fusion but the SVR combination of SVR scores from a single VQI.

Overall, for quality the fusion is very effective and in most cases results in better SROC scores for the fused VQIs than for each individual VQI. A notable exception is the local entropy (LE) VQI as it shows a poor performance in fusion (and regular performance as well). Since we are dealing with encryption the entropy, on paper, sounds like a good way of identifying encryption strength, which usually is tied to visual interference in the image, compare Fig. 1b. In any case, most fusion combinations are superior to the unfused results. Interestingly, the best is not a combination of the best performing VQIs per encoding type, MSSIM (0.9669) and AlexNet (0.8695), but a combination of LSS and AlexNet, SROC of 0.91 which quite substantially improves by 0.1175 on the best single VQI (MSSIM with an SROC score of 0.7956).

For recognition, see Table 4b the same overall observations hold, lots of improvements and the LE VQI being surprisingly weak. The best fusion however is a different one than for quality. The VQIs MSSIM and BLIINDS-II combined exhibit a score of 0.818, which is again quite a strong improvement over the best single VQI performance of 0.732 (MSSIM), compare Table 2. The performance is lower than for quality assessment, but this is hardly a surprise given that VQIs are created to assess quality. The fusion with no-reference metrics (BLIINDS-II) also makes sense for the recognizability rating as the finding of structural elements, as the MSSIM tries, is doomed to fail from a certain encryption strength onward. This makes it all the more surprising that the LE VQI performs so poorly.
<caption>**Table 3**</caption>

Frame score combination by SVR rather than averaging. The resulting SROC scores are presented.

<table><thead><tr><th colspan="3">MJ2k+H.265</th></tr><tr><th>VQI</th><th>Q</th><th>R</th></tr></thead><tbody><tr><td>MSSIM</td><td>0.7925</td><td>0.6973</td></tr><tr><td>LSS</td><td>0.7909</td><td>0.6740</td></tr><tr><td>AlexNet</td><td>0.7786</td><td>0.6873</td></tr><tr><td>PSNR</td><td>0.7557</td><td>0.6287</td></tr><tr><td>LEG</td><td>0.7108</td><td>0.5935</td></tr><tr><td>TOPIQ</td><td>0.6594</td><td>0.4908</td></tr><tr><td>VGG16</td><td>0.6556</td><td>0.6070</td></tr><tr><td>VIF</td><td>0.6445</td><td>0.5340</td></tr><tr><td>CPA1</td><td>0.6194</td><td>0.4930</td></tr><tr><td>ESS</td><td>0.5986</td><td>0.5493</td></tr><tr><td>SSIM</td><td>0.5540</td><td>0.4816</td></tr><tr><td>LFBVS</td><td>0.4312</td><td>0.3987</td></tr><tr><td>BLIINDS-II</td><td>0.4042</td><td>0.4091</td></tr><tr><td>NSD</td><td>0.3721</td><td>0.3400</td></tr><tr><td>BRIQSUE</td><td>0.3635</td><td>0.2290</td></tr><tr><td>LE</td><td>0.1388</td><td>0.2027</td></tr></tbody></table>

<figure><img src="image_13.png" /><figcaption>**Fig. 6.** SROC estimation of quality and recognizability for subsequences of a given length.</figcaption></figure>

### 3.3.1. Impact of sequence length

In the above experiments we used the full sequence length (150 frames or 5 seconds with 29.97 fps) for the assessment of quality and recognition. A question is how many frames are required to get the above results. More is of course more when it comes to assessment but this also holds true for computational requirements. To assess the performance as a function of the number of frames we cut sequences into non-overlapping subsequences of length $N \in \{1, 2, 4, 6, 8, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100\}$ frames. Training regime is the same as above, where leaving one-sequence out means that all subsequences originating from a given sequence are left out. Each subsequence is assigned the same recognition and quality score as the overall sequences, this might in reality not be 100% accurate but it will also not be far off.

Results can be seen in Fig. 6 where the SROC for quality and recognizability are plotted over the subsequence lengths used for evaluation. There are minor variances of course but it is clear that an increased number of frames is beneficial for the estimation of both quality and recognition. This also explains the relatively high performance of VQIs on image sequences compared to reports in literature on still images, e.g., [11]. Still, even with fusion the performances of VQIs as a general tool, i.e., the VQI is not specifically matched to an encoding/encryption scheme (cf. Table 2), is rather low for both quality and recognition.
<caption>**Table 4**</caption>

Score fusion using SVR, with leave on sequence out cross validation, on a combination of encoding types (MJ2k+H.265). Table is mirrored along the prime diagonal, diagonal entries are not fusions but single VQI scores combined by SVR. Entries are marked depending if they are single VQI scores, the combination is better than both VQIs, or best overall.

(a) Spearman rank order correlation with quality on MJ2k+H.265.

<table><thead><tr><th>SROC(Q)</th><th>AlexNet</th><th>BLIINDS-II</th><th>BRISQUE</th><th>CPA1</th><th>ESS</th><th>LEG</th><th>LFBVS</th><th>LSS</th><th>MSSIM</th><th>PSNR</th><th>SSIM</th><th>LE</th><th>TOPIQ</th><th>VGG16</th><th>VIF</th><th>NSD</th></tr></thead><tbody><tr><td>AlexNet</td><td>0.779</td><td>0.842</td><td>0.755</td><td>0.786</td><td>0.723</td><td>0.799</td><td>0.726</td><td><u>0.910</u></td><td>0.863</td><td>0.824</td><td>0.734</td><td>0.652</td><td>0.783</td><td>0.792</td><td>0.784</td><td>0.737</td></tr><tr><td>BLIINDS-II</td><td>0.842</td><td>0.404</td><td>0.557</td><td>0.822</td><td>0.698</td><td>0.791</td><td>0.586</td><td>0.854</td><td>0.901</td><td>0.828</td><td>0.734</td><td>0.267</td><td>0.828</td><td>0.775</td><td>0.790</td><td>0.501</td></tr><tr><td>BRISQUE</td><td>0.755</td><td>0.557</td><td>0.364</td><td>0.628</td><td>0.623</td><td>0.689</td><td>0.561</td><td>0.798</td><td>0.838</td><td>0.767</td><td>0.701</td><td>0.321</td><td>0.667</td><td>0.667</td><td>0.697</td><td>0.705</td></tr><tr><td>CPA1</td><td>0.786</td><td>0.822</td><td>0.628</td><td>0.619</td><td>0.783</td><td>0.724</td><td>0.611</td><td>0.851</td><td>0.817</td><td>0.791</td><td>0.739</td><td>0.608</td><td>0.660</td><td>0.689</td><td>0.720</td><td>0.709</td></tr><tr><td>ESS</td><td>0.723</td><td>0.698</td><td>0.623</td><td>0.783</td><td>0.599</td><td>0.744</td><td>0.627</td><td>0.843</td><td>0.842</td><td>0.808</td><td>0.680</td><td>0.552</td><td>0.748</td><td>0.731</td><td>0.754</td><td>0.697</td></tr><tr><td>LEG</td><td>0.799</td><td>0.791</td><td>0.689</td><td>0.724</td><td>0.744</td><td>0.711</td><td>0.742</td><td>0.790</td><td>0.805</td><td>0.733</td><td>0.760</td><td>0.673</td><td>0.712</td><td>0.735</td><td>0.734</td><td>0.731</td></tr><tr><td>LFBVS</td><td>0.726</td><td>0.586</td><td>0.561</td><td>0.611</td><td>0.627</td><td>0.742</td><td>0.431</td><td>0.872</td><td>0.792</td><td>0.776</td><td>0.651</td><td>0.322</td><td>0.648</td><td>0.600</td><td>0.681</td><td>0.746</td></tr><tr><td>LSS</td><td><u>0.910</u></td><td>0.854</td><td>0.798</td><td>0.851</td><td>0.843</td><td>0.790</td><td>0.872</td><td>0.791</td><td>0.857</td><td>0.834</td><td>0.870</td><td>0.744</td><td>0.820</td><td>0.894</td><td>0.870</td><td>0.865</td></tr><tr><td>MSSIM</td><td>0.863</td><td>0.901</td><td>0.838</td><td>0.817</td><td>0.842</td><td>0.805</td><td>0.792</td><td>0.857</td><td>0.792</td><td>0.812</td><td>0.858</td><td>0.789</td><td>0.801</td><td>0.833</td><td>0.854</td><td>0.806</td></tr><tr><td>PSNR</td><td>0.824</td><td>0.828</td><td>0.767</td><td>0.791</td><td>0.808</td><td>0.733</td><td>0.776</td><td>0.834</td><td>0.812</td><td>0.756</td><td>0.822</td><td>0.738</td><td>0.766</td><td>0.784</td><td>0.796</td><td>0.754</td></tr><tr><td>SSIM</td><td>0.734</td><td>0.734</td><td>0.701</td><td>0.739</td><td>0.680</td><td>0.760</td><td>0.651</td><td>0.870</td><td>0.858</td><td>0.822</td><td>0.554</td><td>0.538</td><td>0.716</td><td>0.710</td><td>0.811</td><td>0.583</td></tr><tr><td>LE</td><td>0.652</td><td>0.267</td><td>0.321</td><td>0.608</td><td>0.552</td><td>0.673</td><td>0.322</td><td>0.744</td><td>0.789</td><td>0.738</td><td>0.538</td><td>0.139</td><td>0.597</td><td>0.558</td><td>0.498</td><td>0.443</td></tr><tr><td>TOPIQ</td><td>0.783</td><td>0.828</td><td>0.667</td><td>0.660</td><td>0.748</td><td>0.712</td><td>0.648</td><td>0.820</td><td>0.801</td><td>0.766</td><td>0.716</td><td>0.597</td><td>0.659</td><td>0.682</td><td>0.752</td><td>0.697</td></tr><tr><td>VGG16</td><td>0.792</td><td>0.775</td><td>0.667</td><td>0.689</td><td>0.731</td><td>0.735</td><td>0.600</td><td>0.894</td><td>0.833</td><td>0.784</td><td>0.710</td><td>0.558</td><td>0.682</td><td>0.656</td><td>0.713</td><td>0.713</td></tr><tr><td>VIF</td><td>0.784</td><td>0.790</td><td>0.697</td><td>0.720</td><td>0.754</td><td>0.734</td><td>0.681</td><td>0.870</td><td>0.854</td><td>0.796</td><td>0.811</td><td>0.498</td><td>0.752</td><td>0.713</td><td>0.645</td><td>0.639</td></tr><tr><td>NSD</td><td>0.737</td><td>0.501</td><td>0.705</td><td>0.709</td><td>0.697</td><td>0.731</td><td>0.746</td><td>0.865</td><td>0.806</td><td>0.754</td><td>0.583</td><td>0.443</td><td>0.697</td><td>0.713</td><td>0.639</td><td>0.372</td></tr></tbody></table>

(b) Recognition Spearman rank order correlation on MJ2k+H.265.

<table><thead><tr><th>SROC(R)</th><th>AlexNet</th><th>BLIINDS-II</th><th>BRISQUE</th><th>CPA1</th><th>ESS</th><th>LEG</th><th>LFBVS</th><th>LSS</th><th>MSSIM</th><th>PSNR</th><th>SSIM</th><th>LE</th><th>TOPIQ</th><th>VGG16</th><th>VIF</th><th>NSD</th></tr></thead><tbody><tr><td>AlexNet</td><td>0.687</td><td>0.739</td><td>0.620</td><td>0.719</td><td>0.629</td><td>0.672</td><td>0.648</td><td>0.806</td><td>0.703</td><td>0.709</td><td>0.658</td><td>0.563</td><td>0.679</td><td>0.714</td><td>0.681</td><td>0.630</td></tr><tr><td>BLIINDS-II</td><td>0.739</td><td>0.409</td><td>0.496</td><td>0.682</td><td>0.620</td><td>0.710</td><td>0.525</td><td>0.738</td><td><u>0.818</u></td><td>0.719</td><td>0.654</td><td>0.353</td><td>0.692</td><td>0.702</td><td>0.647</td><td>0.428</td></tr><tr><td>BRISQUE</td><td>0.620</td><td>0.496</td><td>0.229</td><td>0.473</td><td>0.528</td><td>0.542</td><td>0.472</td><td>0.692</td><td>0.708</td><td>0.616</td><td>0.540</td><td>0.303</td><td>0.467</td><td>0.503</td><td>0.525</td><td>0.573</td></tr><tr><td>CPA1</td><td>0.719</td><td>0.682</td><td>0.473</td><td>0.493</td><td>0.649</td><td>0.615</td><td>0.514</td><td>0.736</td><td>0.695</td><td>0.667</td><td>0.589</td><td>0.408</td><td>0.520</td><td>0.555</td><td>0.578</td><td>0.549</td></tr><tr><td>ESS</td><td>0.629</td><td>0.620</td><td>0.528</td><td>0.649</td><td>0.549</td><td>0.642</td><td>0.569</td><td>0.719</td><td>0.739</td><td>0.707</td><td>0.515</td><td>0.532</td><td>0.608</td><td>0.593</td><td>0.656</td><td>0.647</td></tr><tr><td>LEG</td><td>0.672</td><td>0.710</td><td>0.542</td><td>0.615</td><td>0.642</td><td>0.594</td><td>0.575</td><td>0.663</td><td>0.717</td><td>0.625</td><td>0.612</td><td>0.455</td><td>0.580</td><td>0.611</td><td>0.628</td><td>0.643</td></tr><tr><td>LFBVS</td><td>0.648</td><td>0.525</td><td>0.472</td><td>0.514</td><td>0.569</td><td>0.575</td><td>0.399</td><td>0.736</td><td>0.701</td><td>0.660</td><td>0.554</td><td>0.297</td><td>0.512</td><td>0.473</td><td>0.591</td><td>0.676</td></tr><tr><td>LSS</td><td>0.806</td><td>0.738</td><td>0.692</td><td>0.736</td><td>0.719</td><td>0.663</td><td>0.736</td><td>0.674</td><td>0.749</td><td>0.659</td><td>0.681</td><td>0.638</td><td>0.644</td><td>0.765</td><td>0.683</td><td>0.742</td></tr><tr><td>MSSIM</td><td>0.703</td><td><u>0.818</u></td><td>0.708</td><td>0.695</td><td>0.739</td><td>0.717</td><td>0.701</td><td>0.749</td><td>0.697</td><td>0.706</td><td>0.708</td><td>0.662</td><td>0.692</td><td>0.681</td><td>0.746</td><td>0.733</td></tr><tr><td>PSNR</td><td>0.709</td><td>0.719</td><td>0.616</td><td>0.667</td><td>0.707</td><td>0.625</td><td>0.660</td><td>0.659</td><td>0.706</td><td>0.629</td><td>0.655</td><td>0.542</td><td>0.605</td><td>0.660</td><td>0.630</td><td>0.648</td></tr><tr><td>SSIM</td><td>0.658</td><td>0.654</td><td>0.540</td><td>0.589</td><td>0.515</td><td>0.612</td><td>0.554</td><td>0.681</td><td>0.708</td><td>0.655</td><td>0.482</td><td>0.393</td><td>0.592</td><td>0.607</td><td>0.651</td><td>0.428</td></tr><tr><td>LE</td><td>0.563</td><td>0.353</td><td>0.303</td><td>0.408</td><td>0.532</td><td>0.455</td><td>0.297</td><td>0.638</td><td>0.662</td><td>0.542</td><td>0.393</td><td>0.203</td><td>0.311</td><td>0.449</td><td>0.325</td><td>0.419</td></tr><tr><td>TOPIQ</td><td>0.679</td><td>0.692</td><td>0.467</td><td>0.520</td><td>0.608</td><td>0.580</td><td>0.512</td><td>0.644</td><td>0.692</td><td>0.605</td><td>0.592</td><td>0.311</td><td>0.491</td><td>0.539</td><td>0.609</td><td>0.603</td></tr><tr><td>VGG16</td><td>0.714</td><td>0.702</td><td>0.503</td><td>0.555</td><td>0.593</td><td>0.611</td><td>0.473</td><td>0.765</td><td>0.681</td><td>0.660</td><td>0.607</td><td>0.449</td><td>0.539</td><td>0.607</td><td>0.571</td><td>0.632</td></tr><tr><td>VIF</td><td>0.681</td><td>0.647</td><td>0.525</td><td>0.578</td><td>0.656</td><td>0.628</td><td>0.591</td><td>0.683</td><td>0.746</td><td>0.630</td><td>0.651</td><td>0.325</td><td>0.609</td><td>0.571</td><td>0.534</td><td>0.549</td></tr><tr><td>NSD</td><td>0.630</td><td>0.428</td><td>0.573</td><td>0.549</td><td>0.647</td><td>0.643</td><td>0.676</td><td>0.742</td><td>0.733</td><td>0.648</td><td>0.428</td><td>0.419</td><td>0.603</td><td>0.632</td><td>0.549</td><td>0.340</td></tr></tbody></table>
<figure><img src="image_14.png" /><figcaption>Fig. 7. Decomposition of some encrypted videos into several of their perceptual components.</figcaption></figure>

This middling performance of the VQIs, both still image and video specific ones, are not surprising. The empirical observation of this fact is after all what spurred this research. However, that the dataset presented here shows this behavior means we have properly captured a proper spread of images, which was the intended goal of the data acquisition.

## 3.4. Motion picture constituents under encryption

Seeing that the classical objective quality indices underperforming somewhat, we have decided to decompose the videos into their constituting perceptual components that may (or may not) lead to the recognizability of a given video content. We have opted for a decomposition into four distinct global features that may help the viewer to identify content. Namely, i) the luminance distribution (a smooth dark/bright region within the content), ii) the video motion, which might give away a particular structured movement (people walking/running for instance exhibit a particular motion), iii) some high contrast edges, which could suffer differently from the encryption method and hence maybe be sufficiently preserved to still reveal a particular shape, and iv) a representation of the large smooth portions of the video (captured by a binarization of the sequence). Accordingly, we have computed a temporal averaging ('t-avg'), i.e., the video sequences' frames are stacked and averaged into a single image, the motion (or 'optical flow') [22], we also used a Canny edge detector [1] and a simple Otsu automatic image thresholding [27]. Finally, for each perceptual component,
a similarity score between the reference and distorted sequences was provided thanks to Template Matching [18]. The results are depicted on Fig. 7, which shows for the “crowdrun” video sequence how these four perceptual features (luminance, motion, edges, smooth areas) might be affected by the encryption. The leftmost column shows the observers' subjective scores (both quality and recognizability evaluations). The top row shows the original sequence, and within the black rectangle, the two upper rows present unrecognized sequences, whereas the two lower rows show two easily recognized sequences. Surprisingly enough, for the two unrecognized sequences, both the ‘t’-projection and the binarized sequences provide quite strong similarities with the original video sequence (NCC > 0.9). The strong resemblances between the MJ2k projections (luminances and binary representation) are quite striking although the observers seem to have a hard time making out the content. The temporal smoothing obtained by the projection over time cannot be processed by the human observers, but it provides very strong clues to the objective quality indices when averaging over the frames. Nevertheless, it is also interesting to point out that the somewhat jerky movements (visually quite recognizable) of the runners in the “crowdrun” sequence are quite well captured by the Lucas-Kanade optical flow, while still leading to rather low correlations. The low performance here might come from the similarity measure (template matching) which might not be the best suited to grasp the resemblance between the two motion projections.

Most of the VQIs that have been tested in this work do implicitly consider some of these perceptual components. For the VQIs being specifically designed for quality assessment, i.e., well above the recognition threshold, the matching between the original and the distorted contents is performed locally, i.e., both input contents are compared pixel-wise (or at most block wise). For a recognition-oriented metric to perform well, it would be necessary to conduct this comparison at a more global level. We did an attempt here, using template matching, which worked to some extent. One problem was that the constituents we looked at behaved very differently for different encryption types, so no generally viable solution was found. The other problem is that template matching might be too stringent in that it does not allow some distortion leeway, an example of this are the motion trails of the H.265 encryption in Fig. 7.

## 4. Conclusion

We have presented a database of video sequences which were encrypted with selective encryption methods to produce a wide range of distorted videos ranging from low to high quality as well as from unrecognizable to recognizable content. On this data we collected two kinds of observer scores, a subjective quality and a subjective recognizability score. This data is freely available at the webpage of the WAVELAB group<fnref n="3" /> to reproduce our evaluation or for the development of VQIs for selectively encrypted video content.

We have shown that the video and image quality indices under test, ranging from traditional to CNN based ones, from no reference to full reference and from specific to generic, have not been able to properly cope with the estimation of selective encryption. A combination of metrics (AlexNet combined with the luminance similarity score) using SVR was able to handle quality estimation to a degree (0.91 Spearman rank order correlation with human observers). For recognizability the best combination (MSSIM plus BLINDS-II) only reached an Spearman rank order correlation of 0.82 with human observer scores.

This evaluation shows that (i) current image quality indices are not properly equipped to handle the task of evaluating selectively encrypted content, and (ii) the database we provide the community spans the whole range of selective encryption strength in terms of quality and recognizability. This means it is a good (and currently only available) tool to actually create better VQIs.

We further analyzed the data in terms of constituent structural components, motion trails, edge motion, temporal projection etc. to see if a more video specific approach can be used. The attempts seem promising to a degree but suffer from strong differences between encryption types leading to a doubtful performance as a general purpose tool.

### 4.1. Future works

Future work encompasses three main branches: 1) Extending the database, e.g., to contain more videos with human observer scores to better enable AI based methods, but also to improve it in other aspects. 2) Extend the database to also encompass audio information, as videos rarely come without audio. 3) Use the database to build a better VQI and a recognition index.

Since point 1) is directly related to the topic at hand we will give a bit more detail about possible extensions. The subjective dataset could improved and/or expanded in several ways. For instance, by running a different quality assessment protocol similar to the recognition task. We could include a no-reference quality experiment (the paired comparison being a bit far away from a typical selective encryption scenario). The distorted videos would be presented “as is” and a quality score would be required. With such a protocol, the observers wouldn’t have any high quality reference sequence to help gauge the quality range. Such a subjective protocol might lead to a harsher scoring, and hence a better correlation with recognizability scores. We could also imagine investigating the problem with scene cuts which we mention in section 2. Indeed, a scene cut might give up a lot of information on the recognizability, but if we temporally align two different sequences, so that they have a relatively similar cut (at the exact same time), we might be able to explore the effects of such cuts onto the recognizability. This, however requires access to vast video dataset to pick up some relevant sequences that could be properly temporally aligned, and with possibly the same camera motion. As for the recognizability experiment, we could also imagine having several sequences falling into one recognizability label. We did actually do that with the “office meeting” label being applied to two different sequences (fourpeople and vidyo1), but maybe it would be interesting to have several “town” sequences or “running” videos, acquired at different viewing depth, to see if a given pattern or motion might be easily

<footnote n="3">³ https://wavelab.at/sources/USEE Vid.</footnote>
recognized, despite a different spatial frequency, motion amplitude, etc. induced by the acquisition distance. In other words, a crowd of runners being observed from a distance leads to a different global motion from a few runners being observed up close. It would be quite interesting to study the impact on the subjective quality/recognizability.

### CRediT authorship contribution statement

**Heinz Hofbauer:** Writing – review & editing, Writing – original draft, Visualization, Validation, Software, Resources, Project administration, Methodology, Investigation, Funding acquisition, Formal analysis, Data curation, Conceptualization. **Florent Autrusseau:** Writing – review & editing, Writing – original draft, Visualization, Validation, Software, Resources, Project administration, Methodology, Investigation, Funding acquisition, Formal analysis, Data curation. **Andreas Uhl:** Validation, Supervision, Project administration, Investigation, Funding acquisition, Formal analysis.

### Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

### Data availability

Data will be made available on request.

### Acknowledgements

This work was partially supported by the Austrian Science Fund, project NO. P27776.

### References

[1] J. Canny, A computational approach to edge detection, IEEE Trans. Pattern Anal. Mach. Intell. 8 (1986) 679–698.

[2] M. Carosi, V. Pankajakshan, F. Autrusseau, Towards a simplified perceptual quality metric for watermarking applications, in: Proceedings of SPIE, Multimedia on Mobile Devices, SPIE, San Jose, CA, USA, 2010.

[3] C. Chen, J. Mo, IQA-PyTorch: pytorch toolbox for image quality assessment, [Online]. Available: https://github.com/chaofengc/IQA-PyTorch, 2022.

[4] C. Chen, J. Mo, J. Hou, H. Wu, L. Liao, W. Sun, Q. Yan, W. Lin, Topiq: a top-down approach from semantics to distortions for image quality assessment, IEEE Trans. Image Process. 33 (2024) 2404–2418, https://doi.org/10.1109/TIP.2024.3378466.

[5] S. Cheng, L. Wang, N. Ao, Q. Han, A selective video encryption scheme based on coding characteristics, Symmetry 12 (2020), https://doi.org/10.3390/sym12030332.

[6] L. Dubois, Z. Shahid, W. Puech, Selective Encryption of Images and Videos: From JPEG to H.265/HEVC Through JPEG2000 and H.264/AVC, Nova Publishers, 2014, pp. 137–178.

[7] S. Fossel, G. Fottinger, J. Mohr, Motion JPEG2000 for high quality video systems, IEEE Trans. Consum. Electron. 49 (2003) 787–791, https://doi.org/10.1109/TCE.2003.1261156.

[8] S. Guo, T. Xiang, X. Li, Y. Yang, Peid: a perceptually encrypted image database for visual security evaluation, IEEE Trans. Inf. Forensics Secur. 15 (2020) 1151–1163, https://doi.org/10.1109/TIFS.2019.2935415.

[9] A. Hafsa, M. Fradi, A. Sghaier, J. Malek, M. Machhout, Real-time video security system using chaos-improved advanced encryption standard (iaes), Multimed. Tools Appl. 81 (2022) 2275–2298.

[10] H. Hofbauer, F. Autrusseau, A. Uhl, To recognize or not to recognize — a database of encrypted images with subjective recognition ground truth, Inf. Sci. 551 (2021) 128–145, https://doi.org/10.1016/j.ins.2020.11.047.

[11] H. Hofbauer, F. Autrusseau, A. Uhl, Low quality and recognition of image content, IEEE Trans. Multimed. 24 (2021) 3595–3610, https://doi.org/10.1109/TMM.2021.3103394.

[12] H. Hofbauer, F. Autrusseau, A. Uhl, To recognize or not to recognize - a database of encrypted images with subjective recognition ground truth, Elsevier Inf. Sci. 551 (2021) 128–145.

[13] H. Hofbauer, A. Uhl, An effective and efficient visual quality index based on local edge gradients, in: IEEE 3rd European Workshop on Visual Information Processing, Paris, France, 2011, p. 6.

[14] H. Hofbauer, A. Uhl, Identifying deficits of visual security metrics for images, Signal Process. Image Commun. 46 (2016) 60–75, https://doi.org/10.1016/j.image.2016.05.001.

[15] ISO/IEC 15444-3, Information technology – JPEG2000 image coding system, Part 3: Motion JPEG2000, 2007.

[16] ISO/IEC JCT1/SC29/WG11/N11872, Vision, applications and requirements for high efficiency video coding (HEVC), http://phenix.it-sudparis.eu/mpeg/, 2011.

[17] Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3, J., ISO/IEC JTC1/SC29/WG11, 2012. High efficiency video coding (HEVC) text specification draft 8.

[18] J. Lewis, Fast normalized cross-correlation, in: Vision Interface, Canadian Image Processing and Pattern Recognition Society, 1995, pp. 120–123.

[19] B. Li, W. Zhang, M. Tian, G. Zhai, X. Wang, Blindly assess quality of in-the-wild videos via quality-aware pre-training and motion perception, IEEE Trans. Circuits Syst. Video Technol. 32 (2022) 5944–5958, https://doi.org/10.1109/TCSVT.2022.3164467.

[20] Z. Li, A. Aaron, et al., Toward a practical perceptual video quality metric, Netflix TechBlog, https://netflixtechblog.com/toward-a-practical-perceptual-video-quality-metric-653f208b9652, 2016.

[21] B. Liu, X. Li, H. Yu, J. Lv, A light chaotic encryption algorithm for real-time video encryption, in: 4th EAI International Conference on Robotic Sensor Networks, Springer, 2022, pp. 111–118.

[22] B.D. Lucas, T. Kanade, An iterative image registration technique with an application to stereo vision, in: Proceedings of Imaging Understanding Workshop, 1981, pp. 121–130.

[23] Y. Mao, M. Wu, Security evaluation for communication-friendly encryption of multimedia, in: Proceedings of the IEEE International Conference on Image Processing (ICIP’04), IEEE Signal Processing Society, Singapore, 2004.
[24] X. Min, H. Duan, W. Sun, Y. Zhu, G. Zhai, Perceptual video quality assessment: a survey, arXiv:2402.03413, 2024.

[25] A. Mittal, A.K. Moorthy, A.C. Bovik, No-reference image quality assessment in the spatial domain, IEEE Trans. Image Process. 21 (2012) 4695-4708.

[26] D. Müllner, Modern hierarchical, agglomerative clustering algorithms, arXiv:1109.2378, 2011.

[27] N. Otsu, A threshold selection method from gray-level histograms, IEEE Trans. Syst. Man Cybern. 9 (1979) 62-66.

[28] F. Peng, X. Zhang, Z. Lin, M. Long, A tunable selective encryption scheme for h. 265/hevc based on chroma ipm and coefficient scrambling, IEEE Trans. Circuits Syst. Video Technol. 30 (2020) 2765-2780, https://doi.org/10.1109/TCSVT.2019.2924910.

[29] M. Pinson, S. Wolf, A new standardized method for objectively measuring video quality, IEEE Trans. Broadcast. 50 (2004) 312-322.

[30] C. Qin, J. Hu, F. Li, Z. Qian, X. Zhang, Jpeg image encryption with adaptive dc coefficient prediction and rs pair permutation, IEEE Trans. Multimed. 25 (2023) 2528-2542, https://doi.org/10.1109/TMM.2022.3148591.

[31] M.A. Saad, A.C. Bovik, C. Charrier, Blind image quality assessment: a natural scene statistics approach in the DCT domain, IEEE Trans. Image Process. 21 (2012) 3339-3352.

[32] Z. Shahid, W. Puech, Investigating the structure preserving encryption of high efficiency video coding (HEVC), in: Real-Time Image and Video Processing 2013, 2013, pp. 86560N-86560N-10.

[33] H.R. Sheikh, A.C. Bovik, Image information and visual quality, IEEE Trans. Image Process. 15 (2006) 430-444.

[34] T. Stütz, A. Uhl, On efficient transparent JPEG2000 encryption, in: Proceedings of ACM Multimedia and Security Workshop, MM-SEC '07, ACM, 2007, pp. 97-108.

[35] T. Stütz, A. Uhl, On JPEG2000 error concealment attacks, in: Advantages in Image and Video Technology: Proceedings of the 3rd Pacific-Rim Symposium on Image and Video Technology, PSIVT '09, Springer, 2009, pp. 851-861.

[36] G. Sullivan, J. Ohm, W.J. Han, T. Wiegand, Overview of the High Efficiency Video Coding (HEVC) standard, IEEE Trans. Circuits Syst. Video Technol. 22 (2012) 1649-1668.

[37] J. Sun, Z. Xu, J. Liu, Y. Yao, An objective visual security assessment for cipher-images based on local entropy, Multimed. Tools Appl. (2010), https://doi.org/10.1007/s11042-010-0491-5.

[38] B. Tang, C. Yang, Y. Zhang, A format compliant framework for hevc selective encryption after encoding, IEEE Trans. Circuits Syst. Video Technol. 33 (2023) 1140-1156, https://doi.org/10.1109/TCSVT.2022.3212865.

[39] L. Tong, F. Dai, Y. Zhang, J. Li, Visual security evaluation for video encryption, in: Proceedings of the International Conference on Multimedia, New York, NY, USA, 2010, pp. 835-838.

[40] A. Unterweger, K.V. Ryckegem, D. Engel, A. Uhl, Building a post-compression region-of-interest encryption framework for existing video surveillance systems - challenges, obstacles and practical concerns, Multimed. Syst. 22 (2015) 617-639, https://doi.org/10.1007/s00530-015-0473-6.

[41] Z. Wang, A. Bovik, H. Sheikh, E. Simoncelli, Image quality assessment: from error visibility to structural similarity, IEEE Trans. Image Process. 13 (2004) 600-612.

[42] Z. Wang, E.P. Simoncelli, A.C. Bovik, Multi-scale structural similarity for image quality assessment, in: Proc. 37th IEEE Asilomar Conference on Signals, Systems and Computers, 2003, pp. 1398-1402.

[43] W. Wen, K. Wei, Y. Fang, Y. Zhang, Visual quality assessment for perceptually encrypted light field images, IEEE Trans. Circuits Syst. Video Technol. 31 (2021) 2522-2534, https://doi.org/10.1109/TCSVT.2020.3026817.

[44] H. Wu, E. Zhang, L. Liao, C. Chen, J. Hou, A. Wang, W. Sun, Q. Yan, W. Lin, Exploring video quality assessment on user generated contents from aesthetic and technical perspectives, in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 20144-20154.

[45] T. Xiang, Y. Yang, H. Liu, S. Guo, Visual security evaluation of perceptually encrypted images based on image importance, IEEE Trans. Circuits Syst. Video Technol. 30 (2020) 4129-4142, https://doi.org/10.1109/TCSVT.2019.2955298.

[46] Y. Yang, T. Xiang, H. Liu, X. Liao, Convolutional neural network for visual security evaluation, IEEE Trans. Circuits Syst. Video Technol. 31 (2021) 3293-3307, https://doi.org/10.1109/TCSVT.2020.3036854.

[47] Y. Yao, Z. Xu, J. Sun, Visual security assessment for cipher-images based on neighborhood similarity, Informatica 33 (2009) 69-76.

[48] G. Yue, C. Hou, K. Gu, T. Zhou, H. Liu, No-reference quality evaluator of transparently encrypted images, IEEE Trans. Multimed. 21 (2019) 2184-2194.

[49] G. Zhai, X. Min, Perceptual image quality assessment: a survey, Sci. China Inf. Sci. 63 (2020) 1869-1919, https://doi.org/10.1007/s11432-019-2757-1.

[50] R. Zhang, P. Isola, A.A. Efros, E. Shechtman, O. Wang, The unreasonable effectiveness of deep features as a perceptual metric, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.