In view of the fact that formation of $\omega_{X/Y}^\bullet$ commutes with base change (see Remark \ref{remark-relative-dualizing-complex}), we may and do assume that $Y$ is affine. For a perfect object $E$ of $D(\mathcal{O}_X)$ we have \begin{align*} Rf_*(E \otimes_{\mathcal{O}_X}^\mathbf{L} \omega_{X/Y}^\bullet) & = Rf_*R\SheafHom_{\mathcal{O}_X}(E^\vee, \omega_{X/Y}^\bullet) \\ & = R\SheafHom_{\mathcal{O}_Y}(Rf_*E^\vee, \mathcal{O}_Y) \\ & = (Rf_*E^\vee)^\vee \end{align*} For the first equality, see Cohomology, Lemma \ref{cohomology-lemma-dual-perfect-complex}. For the second equality, see Lemma \ref{lemma-iso-on-RSheafHom}, Remark \ref{remark-iso-on-RSheafHom}, and  Derived Categories of Schemes, Lemma \ref{perfect-lemma-flat-proper-perfect-direct-image-general}. The third equality is the definition of the dual. In particular these references also show that the outcome is a perfect object of $D(\mathcal{O}_Y)$. We conclude that $\omega_{X/Y}^\bullet$ is $Y$-perfect by More on Morphisms, Lemma \ref{more-morphisms-lemma-characterize-relatively-perfect}. This proves (1). \medskip\noindent Let $M$ be an object of $D_\QCoh(\mathcal{O}_Y)$. Then \begin{align*} \Hom_Y(M, Rf_*\omega_{X/Y}^\bullet) & = \Hom_X(Lf^*M, \omega_{X/Y}^\bullet) \\ & = \Hom_Y(Rf_*Lf^*M, \mathcal{O}_Y) \\ & = \Hom_Y(M \otimes_{\mathcal{O}_Y}^\mathbf{L} Rf_*\mathcal{O}_Y, \mathcal{O}_Y) \end{align*} The first equality holds by Cohomology, Lemma \ref{cohomology-lemma-adjoint}. The second equality by construction of $a$. The third equality by Derived Categories of Schemes, Lemma \ref{perfect-lemma-cohomology-base-change}. Recall $Rf_*\mathcal{O}_X$ is perfect of tor amplitude in $[0, N]$ for some $N$, see Derived Categories of Schemes, Lemma \ref{perfect-lemma-flat-proper-perfect-direct-image-general}. Thus we can represent $Rf_*\mathcal{O}_X$ by a complex of finite projective modules sitting in degrees $[0, N]$ (using More on Algebra, Lemma \ref{more-algebra-lemma-perfect} and the fact that $Y$ is affine). Hence if $M = \mathcal{O}_Y[-i]$ for some $i > 0$, then the last group is zero. Since $Y$ is affine we conclude that $H^i(Rf_*\omega_{X/Y}^\bullet) = 0$ for $i > 0$. This proves (2). \medskip\noindent Let $E$ be a perfect object of $D_\QCoh(\mathcal{O}_X)$. Then we have \begin{align*} \Hom_X(E, R\SheafHom_{\mathcal{O}_X}(\omega_{X/Y}^\bullet, \omega_{X/Y}^\bullet) & = \Hom_X(E \otimes_{\mathcal{O}_X}^\mathbf{L} \omega_{X/Y}^\bullet, \omega_{X/Y}^\bullet) \\ & = \Hom_Y(Rf_*(E \otimes_{\mathcal{O}_X}^\mathbf{L} \omega_{X/Y}^\bullet), \mathcal{O}_Y) \\ & = \Hom_Y(Rf_*(R\SheafHom_{\mathcal{O}_X}(E^\vee, \omega_{X/Y}^\bullet)), \mathcal{O}_Y) \\ & = \Hom_Y(R\SheafHom_{\mathcal{O}_Y}(Rf_*E^\vee, \mathcal{O}_Y), \mathcal{O}_Y) \\ & = R\Gamma(Y, Rf_*E^\vee) \\ & = \Hom_X(E, \mathcal{O}_X) \end{align*} The first equality holds by Cohomology, Lemma \ref{cohomology-lemma-internal-hom}. The second equality is the definition of $\omega_{X/Y}^\bullet$. The third equality comes from the construction of the dual perfect complex $E^\vee$, see Cohomology, Lemma \ref{cohomology-lemma-dual-perfect-complex}. The fourth equality follows from the equality $Rf_*R\SheafHom_{\mathcal{O}_X}(E^\vee, \omega_{X/Y}^\bullet) = R\SheafHom_{\mathcal{O}_Y}(Rf_*E^\vee, \mathcal{O}_Y)$ shown in the first paragraph of the proof. The fifth equality holds by double duality for perfect complexes (Cohomology, Lemma \ref{cohomology-lemma-dual-perfect-complex}) and the fact that $Rf_*E$ is perfect by Derived Categories of Schemes, Lemma \ref{perfect-lemma-flat-proper-perfect-direct-image-general}. The last equality is Leray for $f$. This string of equalities essentially shows (3) holds by the Yoneda lemma. Namely, the object $R\SheafHom(\omega_{X/Y}^\bullet, \omega_{X/Y}^\bullet)$ is in $D_\QCoh(\mathcal{O}_X)$ by Derived Categories of Schemes, Lemma \ref{perfect-lemma-quasi-coherence-internal-hom}. Taking $E = \mathcal{O}_X$ in the above we get a map $\alpha : \mathcal{O}_X \to R\SheafHom_{\mathcal{O}_X}(\omega_{X/Y}^\bullet, \omega_{X/Y}^\bullet)$ corresponding to $\text{id}_{\mathcal{O}_X} \in \Hom_X(\mathcal{O}_X, \mathcal{O}_X)$. Since all the isomorphisms above are functorial in $E$ we see that the cone on $\alpha$ is an object $C$ of $D_\QCoh(\mathcal{O}_X)$ such that $\Hom(E, C) = 0$ for all perfect $E$. Since the perfect objects generate (Derived Categories of Schemes, Theorem \ref{perfect-theorem-bondal-van-den-Bergh}) we conclude that $\alpha$ is an isomorphism.