prompt_front: Now, answer the following question based on the text below, only give me the answer and do not output any other words.

Question: What are the stability conditions for a solution of $-\Delta u = f(u)$?

Read the following text and answer briefly.

Context 1: \section{Introduction and main results}


In this note we are interested in the existence versus non-existence of stable sub- and super-solutions of equations of the form
\begin{equation} \label{eq1}
-div( \omega_1(x) \nabla u ) = \omega_2(x) f(u) \qquad \mbox{in $ {\mathbb{R}}^N$,}
\end{equation} where $f(u)$ is one of the following non-linearities: $e^u$,  $ u^p$ where $ p>1$ and $ -u^{-p}$ where $ p>0$   We assume that $ \omega_1(x)$ and $ \omega_2(x)$, which we call \emph{weights},  are smooth positive functions (we allow $ \omega_2$ to be zero at say a point) and which satisfy various growth conditions at $ \infty$     Recall that we say that a solution $ u $ of $ -\Delta u = f(u)$ in $ {\mathbb{R}}^N$ is stable provided
\[ \int f'(u) \psi^2 \le \int | \nabla \psi|^2, \qquad \forall \psi \in C_c^2,\] where $ C_c^2$ is the set of $ C^2$ functions defined on $ {\mathbb{R}}^N$ with compact support   Note that the stability of $u$ is just saying that the second variation at $u$ of the energy associated with the equation is non-negative      In our setting this becomes:  We say a $C^2$ sub/super-solution $u$ of (\ref{eq1}) is \emph{stable} provided
\begin{equation} \label{stable}
\int \omega_2 f'(u) \psi^2 \le \int \omega_1 | \nabla \psi|^2 \qquad \forall \psi \in C_c^2 
\end{equation}
One should note that (\ref{eq1}) can be re-written as
\begin{equation*}
- \Delta u  + \nabla \gamma(x) \cdot \nabla u ={ \omega_2}/{\omega_1}\ f(u) \qquad \text{ in $ \mathbb{R}^N$},
\end{equation*}
where
$\gamma = - \log( \omega_1)$ and on occasion we shall take this point of view 


\begin{remark} \label{triv} Note that  if $ \omega_1$ has enough integrability then it is immediate that if $u$ is a stable solution  of (\ref{eq1}) we have $ \int \omega_2 f'(u) =0 $  (provided $f$ is increasing)    To see this let $ 0 \le \psi \le 1$ be supported in a ball of radius $2R$ centered at the origin ($B_{2R}$) with $ \psi =1$ on $ B_R$ and such that $  | \nabla \psi | \le \frac{C}{R}$ where $ C>0$ is independent of $ R$    Putting this $ \psi$ into $ (\ref{stable})$ one obtains
\[ \int_{B_R} \omega_2 f'(u) \le \frac{C}{R^2} \int_{R < |x| <2R} \omega_1,\] and so if the right hand side goes to zero as $ R \rightarrow \infty$ we have the desired result 

\end{remark}





The existence versus non-existence of stable solutions of  $ -\Delta u =  f(u)$ in $ {\mathbb{R}}^N$ or $ -\Delta u = g(x) f(u)$ in $ {\mathbb{R}}^N$ is now quite well understood, see  \cite{dancer1, farina1, egg, zz, f2, f3, wei, ces, e1, e2}   We remark that some of these results are examining the case where $ \Delta $ is replaced with $ \Delta_p$ (the $p$-Laplacian) and also in many cases the authors are interested in finite Morse index solutions or solutions which are stable outside a compact set 
  Much of the interest in these Liouville type theorems  stems from the fact that the non-existence of a stable solution is related to the existence of a priori estimates for stable solutions of a related equation on a bounded domain 




       In \cite{Ni}  equations similar to  $ -\Delta u = |x|^\alpha u^p$  where examined on the unit ball in $ {\mathbb{R}}^N$ with zero Dirichlet boundary conditions   There it was shown that for $ \alpha >0$ that  one can obtain positive solutions for $ p $ supercritical with respect to Sobolev embedding and so one can view that the term $ |x|^\alpha$ is restoring some compactness       A similar feature happens for equations of the form
\[ -\Delta u = |x|^\alpha f(u) \qquad \mbox{in $ {\mathbb{R}}^N$};\]    the value of $ \alpha$ can vastly alter the existence versus non-existence of a stable solution,  see \cite{e1, ces, e2, zz, egg} 

We now come to our main results and for this we need to define a few quantities:

\begin{eqnarray*}
I_G&:=& R^{-4t-2} \int_{ R < |x|<2R} \frac{ \omega_1^{2t+1}}{\omega_2^{2t}}dx , \\
 J_G&:=& R^{-2t-1} \int_{R < |x| <2R} \frac{| \nabla \omega_1|^{2t+1} }{\omega_2^{2t}} dx  ,\\I_L&:=& R^\frac{-2(2t+p-1)}{p-1}  \int_{R<|x|<2R }{ \left(   \frac{w_1^{p+2t-1}}{w_2^{2t}}    \right)^{\frac{1}{p-1} }  }  dx,\\ J_L&:= &R^{-\frac{p+2t-1}{p-1} }  \int_{R<|x|<2R }{ \left(   \frac{|\nabla w_1|^{p+2t-1}}{w_2^{2t}}     \right)^{\frac{1}{p-1} }  }  dx,\\
I_M  &:=&   R^{-2\frac{p+2t+1}{p+1} }  \int_{R<|x|<2R }{ \left(   \frac{w_1^{p+2t+1}}{w_2^{2t}}    \right)^{\frac{1}{p+1} }  } \ dx, \\
J_M  &:= &  R^{-\frac{p+2t+1}{p+1} }  \int_{R<|x|<2R }{ \left(   \frac{|\nabla w_1|^{p+2t+1}}{w_2^{2t}}     \right)^{\frac{1}{p+1} }  }  dx 
\end{eqnarray*}


The three equations we examine are
\[ -div( \omega_1 \nabla u ) = \omega_2 e^u \qquad \mbox{ in $ {\mathbb{R}}^N$ } \quad  (G), \]
\[  -div( \omega_1 \nabla u ) = \omega_2 u^p \qquad \mbox{ in $ {\mathbb{R}}^N$ } \quad  (L), \]
\[ -div( \omega_1 \nabla u ) = - \omega_2 u^{-p} \qquad \mbox{ in $ {\mathbb{R}}^N$ } \quad  (M),\]  and where we restrict $(L)$ to the case $ p>1$ and $(M)$ to $ p>0$     By solution we always mean a $C^2$ solution    We now come to our main results in terms of abstract $ \omega_1 $ and $ \omega_2$     We remark that our approach to non-existence of stable solutions is the approach due to Farina, see \cite{f2,f3,farina1} 

\begin{thm} \label{main_non_exist} \begin{enumerate}


\item   There is no  stable sub-solution of $(G)$ if $ I_G, J_G \rightarrow 0$ as $ R \rightarrow \infty$   for some   $0<t<2$ 
 \item    There is no positive   stable sub-solution (super-solution) of $(L)$  if  $ I_L,J_L \rightarrow 0$ as $ R \rightarrow \infty$   for some   $p- \sqrt{p(p-1)}   < t<p+\sqrt{p(p-1)} $  ($0<t<\frac{1}{2}$) 
\item   There is no positive   stable super-solution of (M) if  $ I_M,J_M \rightarrow 0$ as $ R \rightarrow \infty$  for some $0<t<p+\sqrt{p(p+1)}$ 

\end{enumerate}

\end{thm}     If we assume that $ \omega_1$ has some monotonicity we can do  better    We will assume that the monotonicity conditions is satisfied for big $x$  but really all ones needs is for it to be satisfied on a suitable sequence of annuli 

\begin{thm} \label{mono}  \begin{enumerate} \item There is no  stable sub-solution of $(G)$ with $  \nabla \omega_1(x) \cdot  x \le 0$ for big $x$ if  $ I_G \rightarrow 0$ as $ R \rightarrow \infty$   for some   $0<t<2$ 

\item There is no positive  stable sub-solution of  $(L)$ provided  $  I_L \rightarrow 0$ as $ R \rightarrow \infty$ for either:
\begin{itemize}
\item  some $  1 \le t < p + \sqrt{p(p-1)}$ and $  \nabla \omega_1(x) \cdot x \le 0$ for big $x$,  or  \\

\item some $ p - \sqrt{p(p-1)} < t \le 1$ and $ \nabla \omega_1(x) \cdot x \ge 0$ for big $ x$ 
\end{itemize}
 There is no positive super-solution of $(L)$ provided $ I_L \rightarrow 0$ as $ R \rightarrow \infty$ for some $ 0 < t < \frac{1}{2}$ and $ \nabla \omega_1(x) \cdot x \le 0$ for big $x$ 



\item  There is no positive stable super-solution of $(M)$ provided  $ I_M \rightarrow 0$ as $ R \rightarrow \infty$  for some  $0<t<p+\sqrt{p(p+1)}$ 

\end{enumerate}

\end{thm}


\begin{cor}  \label{thing}  Suppose $ \omega_1 \le C  \omega_2$  for big $ x$, $ \omega_2 \in L^\infty$,  $ \nabla \omega_1(x) \cdot  x \le 0$ for big $ x$ 
\begin{enumerate} \item There is no stable sub-solution of $(G)$ if $ N \le 9$ 

\item  There is no positive stable sub-solution of $(L)$ if  $$N<2+\frac{4}{p-1} \left( p+\sqrt{p(p-1)}  \right) $$

\item  There is no positive stable super-solution of $(M)$ if $$N<2+\frac{4}{p+1} \left( p+\sqrt{p(p+1)}  \right) $$

\end{enumerate}

\end{cor}


If one takes $ \omega_1=\omega_2=1$ in the above corollary, the results obtained for $(G)$ and  $(L)$,  and for some values of $p$ in $(M)$, are optimal, see \cite{f2,f3,zz} 





We now drop all monotonicity conditions on $ \omega_1$ 

\begin{cor} \label{po} Suppose  $ \omega_1 \le C \omega_2$ for big $x$, $ \omega_2 \in L^\infty$, $ | \nabla \omega_1| \le C \omega_2$ for big $x$ 
\begin{enumerate} \item  There is no stable sub-solution of $(G)$ if $ N \le 4$ 

\item  There is no positive stable sub-solution of $(L)$ if $$N<1+\frac{2}{p-1} \left( p+\sqrt{p(p-1)}  \right) $$

\item There is no positive super-solution of $(M)$ if $$N<1+\frac{2}{p+1} \left( p+\sqrt{p(p+1)}  \right) $$

\end{enumerate}

\end{cor}

Some of the conditions on $ \omega_i$ in Corollary \ref{po} seem somewhat artificial   If we shift over to the advection equation (and we take $ \omega_1=\omega_2$  for simplicity)
\[ -\Delta u + \nabla \gamma \cdot \nabla u = f(u), \] the conditions on $ \gamma$ become: $ \gamma$ is bounded from below and has a bounded gradient 



In what follows we examine the case where $ \omega_1(x) = (|x|^2 +1)^\frac{\alpha}{2}$ and $ \omega_2(x)= g(x) (|x|^2 +1)^\frac{\beta}{2}$,  where $ g(x) $ is positive except at say a point, smooth and where $ \lim_{|x| \rightarrow \infty} g(x) = C \in (0,\infty)$      For this class of weights we can essentially obtain optimal results 


\begin{thm} \label{alpha_beta}   Take $ \omega_1 $ and $ \omega_2$ as above 
\begin{enumerate}

\item If $ N+ \alpha - 2 <0$ then there is no stable sub-solution for $(G)$, $(L)$ (here we require it to be positive) and in the case of $(M)$ there is no positive  stable  super-solution  This case is the trivial case, see Remark \ref{triv}   \\



\textbf{Assumption:} For the remaining cases we assume that $ N + \alpha -2 > 0$ 

  \item If  $N+\alpha-2<4(\beta-\alpha+2)$ then there is no  stable sub-solution for $ (G)$ 

\item If $N+\alpha-2<\frac{ 2(\beta-\alpha+2)   }{p-1} \left( p+\sqrt{p(p-1)}  \right)$ then there is  no positive stable sub-solution of $(L)$ 

\item If $N+\alpha-2<\frac{2(\beta-\alpha+2)   }{p+1} \left( p+\sqrt{p(p+1)}  \right)$ then there is no positive stable super-solution of $(M)$ 

\item Further more 2,3,4 are optimal in the sense if $ N + \alpha -2 > 0$ and the remaining inequality is not satisfied (and in addition we assume we don't have equality in the inequality) then we can find a suitable function $ g(x)$ which satisfies the above properties and a stable sub/super-solution $u$ for the appropriate equation 

\end{enumerate}

\end{thm}

\begin{remark}  Many of the above results can be extended to the case of equality in either the $ N + \alpha - 2 \ge 0$ and also the other inequality which depends on the equation we are  examining   We omit the details because one cannot prove the results in a unified way 

\end{remark}






















In showing that an explicit solution is stable we will need the   weighted Hardy inequality given in \cite{craig} 
\begin{lemma} \label{Har}
Suppose $ E>0$ is a smooth function   Then one has
\[ (\tau-\frac{1}{2})^2 \int E^{2\tau-2} | \nabla E|^2 \phi^2 + (\frac{1}{2}-\tau) \int (-\Delta E) E^{2\tau-1} \phi^2 \le \int E^{2\tau} | \nabla \phi|^2,\] for all $ \phi \in C_c^\infty({\mathbb{R}}^N)$ and $ \tau \in {\mathbb{R}}$ 
\end{lemma}  By picking an appropriate function $E$ this gives,

\begin{cor} \label{Hardy}
For all $ \phi \in C_c^\infty$ and $ t , \alpha \in {\mathbb{R}}$  We have
  \begin{eqnarray*}
\int (1+|x|^2)^\frac{\alpha}{2} |\nabla\phi|^2 &\ge& (t+\frac{\alpha}{2})^2 \int |x|^2 (1+|x|^2)^{-2+\frac{\alpha}{2}}\phi^2\\
&&+(t+\frac{\alpha}{2})\int  (N-2(t+1)   \frac{|x|^2}{1+|x|^2}) (1+|x|^2)^{-1+\frac{\alpha} {2}} \phi^2 
\end{eqnarray*}
 \end{cor}























\section{Proof of main results}

\textbf{ Proof of Theorem \ref{main_non_exist} }   (1)  Suppose $ u$ is a stable sub-solution of $(G)$ with $ I_G,J_G \rightarrow 0$ as $ R \rightarrow \infty$ and  let $ 0 \le \phi \le 1$ denote a smooth compactly supported function   Put $ \psi:= e^{tu} \phi$ into (\ref{stable}), where $ 0 <t<2$, to arrive at
\begin{eqnarray*}
\int \omega_2 e^{(2t+1)u} \phi^2 &\le & t^2 \int \omega_1 e^{2tu} | \nabla u|^2 \phi^2 \\
&& +\int \omega_1 e^{2tu}|\nabla \phi|^2 + 2 t \int \omega_1 e^{2tu} \phi \nabla u \cdot \nabla \phi 
\end{eqnarray*}  Now multiply $(G)$ by $ e^{2tu} \phi^2$ and integrate by parts to arrive at
\[ 2t \int \omega_1 e^{2tu} | \nabla u|^2 \phi^2 \le \int \omega_2 e^{(2t+1) u} \phi^2 - 2 \int \omega_1 e^{2tu} \phi \nabla u \cdot \nabla \phi,\]
and now if one equates like terms they arrive at
\begin{eqnarray} \label{start}
\frac{(2-t)}{2} \int \omega_2 e^{(2t+1) u} \phi^2  & \le & \int \omega_1 e^{2tu} \left( | \nabla \phi |^2 - \frac{ \Delta \phi}{2} \right) dx \nonumber \\
&& - \frac{1}{2} \int e^{2tu} \phi \nabla \omega_1 \cdot \nabla \phi 
\end{eqnarray}   Now substitute $ \phi^m$ into this inequality for $ \phi$ where $ m $ is a big integer to obtain
\begin{eqnarray} \label{start_1}
\frac{(2-t)}{2} \int \omega_2 e^{(2t+1) u} \phi^{2m}  & \le & C_m \int \omega_1 e^{2tu} \phi^{2m-2}  \left( | \nabla \phi |^2 + \phi |\Delta \phi|  \right) dx \nonumber \\
&& - D_m \int e^{2tu} \phi^{2m-1} \nabla \omega_1 \cdot \nabla \phi
\end{eqnarray} where $ C_m$ and $ D_m$ are positive constants just depending on $m$    We now estimate the  terms on the right but we mention that when ones assume the appropriate monotonicity on $ \omega_1$ it is the last integral on the right which one is able to drop 

\begin{eqnarray*}
\int \omega_1 e^{2tu} \phi^{2m-2} | \nabla \phi|^2 & = & \int \omega_2^\frac{2t}{2t+1} e^{2tu} \phi^{2m-2}  \frac{ \omega_1 }{\omega_2^\frac{2t}{2t+1}} | \nabla \phi|^2  \\
& \le &  \left( \int \omega_2 e^{(2t+1) u} \phi^{(2m-2) \frac{(2t+1)}{2t}} dx \right)^\frac{2t}{2t+1}\\ &&\left( \int \frac{ \omega_1 ^{2t+1}}{\omega_2^{2t}} | \nabla \phi |^{2(2t+1)} \right)^\frac{1}{2t+1} 
\end{eqnarray*}
Now, for fixed $ 0 <t<2$ we can take $ m $ big enough so $ (2m-2) \frac{(2t+1)}{2t} \ge 2m $ and since $ 0 \le \phi \le 1$ this allows us to replace the power on $ \phi$ in the first term on the right with $2m$   and hence we obtain
 \begin{equation} \label{three}
 \int \omega_1 e^{2tu} \phi^{2m-2} | \nabla \phi|^2  \le \left( \int \omega_2 e^{(2t+1) u} \phi^{2m} dx \right)^\frac{2t}{2t+1} \left( \int \frac{ \omega_1 ^{2t+1}}{\omega_2^{2t}} | \nabla \phi |^{2(2t+1)} \right)^\frac{1}{2t+1} 
 \end{equation}    We now take the test functions $ \phi$ to be such that $ 0 \le  \phi \le 1$ with $ \phi $ supported in the ball $ B_{2R}$ with $ \phi = 1 $ on $ B_R$ and $ | \nabla \phi | \le \frac{C}{R}$ where $ C>0$ is independent of $ R$    Putting this choice of $ \phi$ we obtain
 \begin{equation} \label{four}
 \int \omega_1 e^{2tu} \phi^{2m-2} | \nabla \phi |^2 \le \left( \int \omega_2 e^{(2t+1)u} \phi^{2m} \right)^\frac{2t}{2t+1} I_G^\frac{1}{2t+1} 
 \end{equation}  One similarly shows that
 \[ \int \omega_1 e^{2tu} \phi^{2m-1} | \Delta \phi| \le \left( \int \omega_2 e^{(2t+1)u} \phi^{2m} \right)^\frac{2t}{2t+1} I_G^\frac{1}{2t+1} \]
 So, combining the results we obtain

 \begin{eqnarray} \label{last} \nonumber \frac{(2-t)}{2} \int \omega_2 e^{(2t+1) u} \phi^{2m} &\le& C_m \left( \int \omega_2 e^{(2t+1) u} \phi^{2m} dx \right)^\frac{2t}{2t+1} I_G^\frac{1}{2t+1}\\
 &&- D_m \int e^{2tu} \phi^{2m-1}  \nabla \omega_1 \cdot \nabla \phi 
 \end{eqnarray}
 We now estimate this last term   A similar argument using H\"{o}lder's inequality shows that
 \[ \int e^{2tu} \phi^{2m-1} | \nabla \omega_1| | \nabla \phi| \le \left(  \int \omega_2 \phi^{2m} e^{(2t+1) u} dx \right)^\frac{2t}{2t+1} J_G^\frac{1}{2t+1}  \] Combining the results gives that
\begin{equation} \label{last}
(2-t) \left( \int \omega_2 e^{(2t+1) u} \phi^{2m} dx \right)^\frac{1}{2t+1} \le I_G^\frac{1}{2t+1} + J_G^\frac{1}{2t+1},
\end{equation} and now we send $ R \rightarrow \infty$ and use the fact that $ I_G, J_G \rightarrow 0$ as $ R \rightarrow \infty$ to see that
\[ \int \omega_2 e^{(2t+1) u} =0, \] which is clearly a contradiction    Hence there is no stable sub-solution of $(G)$ 

(2)    Suppose that $u >0$ is a stable sub-solution (super-solution) of $(L)$   Then a similar calculation as in (1) shows that for  $ p - \sqrt{p(p-1)} <t < p + \sqrt{p(p-1)}$,  $( 0 <t<\frac{1}{2})$ one has

\begin{eqnarray}   \label{shit}
(p  -\frac{t^2}{2t-1}   )\int \omega_2 u^{2t+p-1} \phi^{2m} & \le & D_m \int \omega_1 u^{2t} \phi^{2(m-1)} (|\nabla\phi|^2  +\phi |\Delta \phi |) \nonumber \\
&& +C_m \frac{(1-t)}{2(2t-1)} \int u^{2t} \phi^{2m-1}\nabla \omega_1 \cdot  \nabla \phi 
 \end{eqnarray}    One now applies H\"{o}lder's argument as in (1) but the terms $ I_L$ and $J_L$ will appear on the right hand side of the resulting
 equation       This shift from a sub-solution to a super-solution depending on whether $ t >\frac{1}{2}$ or $ t < \frac{1}{2}$ is a result from the sign change of $ 2t-1$ at $ t = \frac{1}{2}$    We leave the details for the reader 


(3)   This case is also similar to (1) and (2) 


\hfill $ \Box$

 \textbf{Proof of Theorem \ref{mono} }   (1)   Again we suppose there is a stable sub-solution $u$ of $(G)$   Our starting point  is (\ref{start_1}) and we wish to be able to drop the term
 \[ - D_m \int e^{2tu} \phi^{2m-1} \nabla \omega_1 \cdot \nabla \phi, \] from (\ref{start_1})     We can choose $ \phi$ as in the proof of Theorem \ref{main_non_exist} but also such that $ \nabla \phi(x) = - C(x) x$ where $ C(x) \ge 0$      So if we assume that $ \nabla \omega_1 \cdot x \le 0$ for big $x$ then we see that this last term is non-positive and hence we can drop the term   The the proof is as before but now we only require that $ \lim_{R \rightarrow \infty} I_G=0$ 

 (2)      Suppose that $ u >0$ is a stable sub-solution of $(L)$  and so (\ref{shit}) holds for all $  p - \sqrt{p(p-1)} <t< p + \sqrt{p(p-1)}$    Now we wish to use monotonicity to drop the term from (\ref{shit}) involving the term $ \nabla \omega_1 \cdot \nabla \phi$       $ \phi$ is chosen the same as in (1)  but here one notes that the co-efficient for this term changes sign at $ t=1$ and hence by restriction $t$ to the appropriate side of 1 (along with the above condition on $t$ and $\omega_1$) we can drop the last term depending on which monotonicity we have and hence to obtain a contraction we only require that $ \lim_{R \rightarrow \infty} I_L =0$    The result for the non-existence of a stable super-solution is similar be here one restricts $ 0 < t < \frac{1}{2}$ 


(3)   The proof here is similar to (1) and (2) and we omit the details 




 \hfill $\Box$













\textbf{Proof of Corollary \ref{thing} }   We suppose  that $ \omega_1 \le C  \omega_2$  for big $ x$, $ \omega_2 \in L^\infty$,  $ \nabla \omega_1(x) \cdot  x \le 0$ for big $ x$      \\
(1)     Since $ \nabla \omega_1 \cdot x \le 0$ for big $x$ we can apply Theorem \ref{mono} to show the non-existence of a stable solution to $(G)$    Note that  with the above assumptions on $ \omega_i$ we have that
\[ I_G \le \frac{C R^N}{R^{4t+2}} \]  For $ N \le 9$  we can take $ 0 <t<2$  but close enough to $2$ so the right hand side goes to zero as $ R \rightarrow \infty$ 

Both (2) and (3) also follow directly from applying Theorem \ref{mono}    Note that one can say more about (2) by taking the multiple cases as listed in Theorem \ref{mono} but we have choice to leave this to the reader 


\hfill $ \Box$


\textbf{Proof of Corollary \ref{po} }   Since we have no monotonicity conditions now we will need both $I$ and $J$ to go to zero to show the non-existence of a stable solution    Again the results are obtained immediately by applying Theorem \ref{main_non_exist}  and we prefer to omit the details 


\hfill $\Box$
















\textbf{Proof of Theorem \ref{alpha_beta} }  (1)   If $ N + \alpha -2 <0$ then using Remark \ref{triv}  one easily sees there is no stable sub-solution of $(G)$ and $(L)$ (positive for $(L)$) or a positive stable super-solution of $(M)$    So we now assume that $ N + \alpha -2 > 0$      Note that the monotonicity of $ \omega_1$ changes when $ \alpha $ changes sign and hence one would think that we need to consider separate cases if we hope to utilize the monotonicity results    But a computation shows that in fact $ I$ and $J$ are just multiples of each other in all three cases so it suffices to show, say, that $ \lim_{R \rightarrow \infty} I =0$  \\
(2)    Note that for $ R >1$ one has
\begin{eqnarray*}
I_G & \le  & \frac{C}{R^{4t+2}} \int_{R <|x| < 2R} |x|^{ \alpha (2t+1) - 2t \beta} \\
& \le  &  \frac{C}{R^{4t+2}}  R^{N + \alpha (2t+1) - 2t \beta},
\end{eqnarray*} and so to show the non-existence we want to find some $ 0 <t<2$ such that
$  4t+2 > N  + \alpha(2t+1) - 2 t \beta$,   which is equivalent to  $ 2t ( \beta - \alpha +2) > (N + \alpha -2)$     Now recall that we are assuming that $ 0 < N + \alpha -2 < 4 ( \beta - \alpha +2) $ and hence we have the desired result by taking $  t <2$ but sufficiently close 
The proof of the non-existence results for
(3) and (4) are similar and we omit the details    \\
(5)   We now assume that $N+\alpha-2>0$   In showing the existence of stable sub/super-solutions we need to consider  $ \beta - \alpha + 2 <0$ and $ \beta - \alpha +2 >0$ separately 

\begin{itemize} \item $(\beta - \alpha + 2 <0)$  Here we take $ u(x)=0$ in the case of $(G)$ and $ u=1$ in the case of $(L)$ and $(M)$  In addition we take $ g(x)=\E$   It is clear that in all cases $u$ is the appropriate sub or super-solution   The only thing one needs to check is the stability     In all cases this reduces to trying to show that we have
\[ \sigma \int       (1+|x|^2)^{\frac{\alpha}{2}  -1}    \phi^2 \le \int      (1+|x|^2)^{\frac{\alpha}{2}}   | \nabla\phi  |^2,\]  for all $ \phi \in C_c^\infty$ where  $ \sigma $ is some small positive constant; its either $ \E$ or $ p \E$ depending on which equation were are examining 
To show this we use the result from Corollary \ref{Hardy} and we drop a few positive terms to arrive at
\begin{equation*}
\int (1+|x|^2)^\frac{\alpha}{2} |\nabla\phi|^2\ge (t+\frac{\alpha}{2})\int \left (N-2(t+1)   \frac{|x|^2}{1+|x|^2}\right) (1+|x|^2)^{-1+\frac{\alpha} {2}}
\end{equation*} which holds for all $ \phi \in C_c^\infty$ and $ t,\alpha \in {\mathbb{R}}$ 
  Now, since $N+\alpha-2>0$, we can choose $t$ such that $-\frac{\alpha}{2}<t<\frac{n-2}{2}$   So, the integrand function in the right hand side is positive and since for small enough $\sigma$ we have
  \begin{equation*}
\sigma \le  (t+\frac{\alpha}{2})(N-2(t+1)   \frac{|x|^2}{1+|x|^2})  \ \ \ \text {for all} \ \ x\in \mathbb{R}^N
\end{equation*}
 we get stability 



\item ($\beta-\alpha+2>0$) In the case of $(G)$ we take   $u(x)=-\frac{\beta-\alpha+2}{2} \ln(1+|x|^2)$ and $g(x):= (\beta-\alpha+2)(N+(\alpha-2)\frac{|x|^2}{1+|x|^2})$  By a computation one sees that $u$ is a sub-solution of $(G)$ and hence we need now to only show the stability, which amounts to showing that
\begin{equation*}
\int \frac{g(x)\psi^2}{(1+|x|^{2   })^{-\frac{\alpha}{2}+1}}\le \int\frac{|\nabla\psi|^2}{    (1+|x|^2)^{-\frac{\alpha}{2}}     },
\end{equation*} for all $ \psi \in C_c^\infty$   To show this we use  Corollary \ref{Hardy}   So  we  need to choose an appropriate $t$ in   $-\frac{\alpha}{2}\le t\le\frac{N-2}{2}$  such that for all $x\in {\mathbb{R}}^N$ we have
 \begin{eqnarray*}
 (\beta-\alpha+2)\left(    N+  (\alpha-2)\frac{|x|^2}{1+|x|^2}\right)         &\le& (t+\frac{\alpha}{2})^2 \frac{ |x|^2 }{(1+|x|^2}\\
&&+(t+\frac{\alpha}{2}) \left(N-2(t+1)   \frac{|x|^2}{1+|x|^2}\right) 
\end{eqnarray*}
With a  simple calculation one sees we need just to have
   \begin{eqnarray*}
 (\beta-\alpha+2)&\le& (t+\frac{\alpha}{2}) \\
  (\beta-\alpha+2)     \left(    N+  \alpha-2\right)      &   \le&  (t+\frac{\alpha}{2}) \left(N-t-2+\frac{\alpha}{2}) \right) 
 \end{eqnarray*}     If one takes $ t= \frac{N-2}{2}$ in the case where $ N \neq 2$ and $ t $ close to zero in the case for $ N=2$ one easily sees the above inequalities both hold, after considering all the constraints on $ \alpha,\beta$ and $N$ 

 We now consider the case of $(L)$   Here one takes $g(x):=\frac {\beta-\alpha+2}{p-1}(    N+  (\alpha-2-\frac{\beta-\alpha+2}{p-1})
\frac{|x|^2}{1+|x|^2})$ and $ u(x)=(1+|x|^2)^{ -\frac  {\beta-\alpha+2}{2(p-1)} }$    Using essentially the same approach as in $(G)$ one shows that $u$ is a stable sub-solution of $(L)$ with this choice of $g$    \\
For the case of $(M)$ we take   $u(x)=(1+|x|^2)^{ \frac  {\beta-\alpha+2}{2(p+1)}   }$ and $g(x):=\frac {\beta-\alpha+2}{p+1}(    N+  (\alpha-2+\frac{\beta-\alpha+2}{p+1})
\frac{|x|^2}{1+|x|^2})$ 



\end{itemize}


\hfill  $  \Box$









Context 2: Paper Info

Title: Efficient nonparametric estimation of Toeplitz covariance matrices
Publish Date: March 20, 2023
Author List: Karolina Klockmann (from Department of Statistics and Operations Research, Universität Wien), Tatyana Krivobokova (from Department of Statistics and Operations Research, Universität Wien)

Figure

Figure 1: Spectral density functions (first row) and autocovariance functions (second row) for examples 1, 2, 3 
Figure 2: Distance between the first atom and the first center of mass of aquaporin (left) and the opening diameter y t over time t (right) 
black line in the left plot) confirms that the covariance matrix estimated with our VST-DCT method almost completely decorrelates the channel diameter Y on the training data set Next, we estimated the regression coefficients β with the usual PLS algorithm, ignoring the dependence in the data Finally, we estimated β with PLS that takes into account dependence using our covariance estimator Σ Based on these regression coefficient estimators, the prediction on the test set was calculated The plot on the right side of Figure 2 shows the Pearson correlation between the true channel diameter on the test set and the prediction on the same test set based on raw (grey) and decorrelated data (black) 
Figure 3: On the left, the auto-correlation function of Y (grey) and of Σ−1/2 Y (black), where Σ is estimated with the VST-DCT method; On the right, correlation between the true values on the test data set and prediction based on partial least squares (in grey) and corrected partial least squares (black) 
Uniform distributionThe observations follow a uniform distribution with covariance matrices Σ 1 , Σ 2 , Σ 3 of examples 1, 2, 3, i e , Y i = Σ 1/2 j X i , j = 1, 3, with X 1 ,    the parameter innov of the R function arima sim is used to pass the innovations X 1 ,    , X n i i d Table4, 5 and 6 show respectively the results for (A) p = 5000, n = 1, (B) p = 1000, n = 50 and (C) p = 5000, n = 10 
(A) p = 5000, n = 1: Errors of the Toeplitz covariance matrix and the spectral density estimators with respect to the spectral and L 2 norm, respectively, as well as the average computation time of the covariance estimators in seconds for one Monte Carlo sample (last column) 
(C) p = 5000, n = 10: Errors of the Toeplitz covariance matrix and the spectral density estimators with respect to the spectral and L 2 norm, respectively, as well as the average computation time of the covariance estimators in seconds for one Monte Carlo sample (last column) 
(A) p = 5000, n = 1: Errors of the Toeplitz covariance matrix and the spectral density estimators with respect to the spectral norm and the L 2 norm, respectively Average computation time of the covariance estimators in seconds for one Monte Carlo sample (last column) 
(B) p = 1000, n = 50: Errors of the Toeplitz covariance matrix and the spectral density estimators with respect to the spectral norm and the L 2 norm, respectively Average computation time of the covariance estimators in seconds for one Monte Carlo sample (last column) 
(C) p = 5000, n = 10: Errors of the Toeplitz covariance matrix and the spectral density estimators with respect to the spectral norm and the L 2 norm, respectively Average computation time of the covariance estimators in seconds for one Monte Carlo sample (last column) 

abstract

A new nonparametric estimator for Toeplitz covariance matrices is proposed  This estimator is based on a data transformation that translates the problem of Toeplitz covariance matrix estimation to the problem of mean estimation in an approximate Gaussian regression  The resulting Toeplitz covariance matrix estimator is positive definite by construction, fully data-driven and computationally very fast 
Moreover, this estimator is shown to be minimax optimal under the spectral norm for a large class of Toeplitz matrices  These results are readily extended to estimation of inverses of Toeplitz covariance matrices  Also, an alternative version of the Whittle likelihood for the spectral density based on the Discrete Cosine Transform (DCT) is proposed 
The method is implemented in the R package vstdct that accompanies the paper 

Introduction

Estimation of covariance and precision matrices is a fundamental problem in statistical data analysis with countless applications in the natural and social sciences  Covariance matrices with a Toeplitz structure arise in the study of stationary stochastic n = 1, to the best of our knowledge, there is no fully data-driven approach for selecting the banding/tapering/thresholding parameter 
suggested first to split the time series into non-overlapping subseries and then apply the cross-validation criterion of   However, it turns out that the right choice of the subseries length is crucial for this approach, but there is no data-based method available for this  In this work, an alternative way to estimate a Toeplitz covariance matrix and its inverse is chosen 
Our approach exploits the one-to-one correspondence between Toeplitz covariance matrices and their spectral densities  First, the given data are transformed into approximate Gaussian random variables whose mean equals to the logarithm of the spectral density  Then, the log-spectral density is estimated by a periodic smoothing spline with a data-driven smoothing parameter 
Finally, the resulting spectral density estimator is transformed into an estimator for Σ or its inverse  It is shown that this procedure leads to an estimator that is fully data-driven, automatically positive definite and achieves the minimax optimal convergence rate under the spectral norm over a large class of Toeplitz covariance matrices 
In particular, this class includes Toeplitz covariance matrices that correspond to long-memory processes with bounded spectral densities  Moreover, the computation is very efficient, does not require iterative or resampling schemes and allows to apply any inference and adaptive estimation procedures developed in the context of nonparametric Gaussian regression 
Estimation of the spectral density from a stationary time series is a research topic with a long history  Earlier nonparametric methods are based on smoothing of the (log-)periodogram, which itself is not a consistent estimator   Another line of nonparametric methods for estimating the spectral density is based on the Whittle likelihood, which is an ap-proximation to the exact likelihood of the time series in the frequency domain 
For example, estimated the spectral density from a penalized Whittle likelihood, while used polynomial splines to estimate the log-spectral density function maximizing the Whittle likelihood  Recently, Bayesian methods for spectral density estimation have been proposed (see , but these may become very computationally intensive in large samples due to posterior sampling 
The minimax optimal convergence rate for nonparametric estimators of Hölder continuous spectral densities from Gaussian stationary time series was obtained by under the L p , 1 ≤ p ≤ ∞, norm  Only few works on spectral density estimation show the optimality of the corresponding estimators  In particular, and derived convergence rates of their estimators for the log-spectral density under the L 2 norm, while neglecting the Whittle likelihood approximation error 
In general, most works on spectral density estimation do not exploit further the close connection to the corresponding Toeplitz covariance matrix estimation  In particular, an upper bound for the L ∞ risk of a spectral density estimator automatically provides an upper bound for the risk of the corresponding Toeplitz covariance matrix estimator under the spectral norm 
This fact is used to establish the minimax optimality of our nonparametric estimator for the Toeplitz covariance matrices  The main contribution of this work is to show that our proposed spectral density estimator is not only numerically very efficient, but also achieves the minimax optimal rate in the L ∞ norm, which in turn ensures the minimax optimality of the corresponding Toeplitz covariance matrix estimator 
The paper is structured as follows  In Section 2, the model is introduced and ap-proximate diagonalization of Toeplitz covariance matrices with the discrete cosine transform is discussed  Moreover, an alternative version of the Whittle's likelihood is proposed  In Section 3, new estimators for the Toeplitz covariance matrix and the precision matrix are derived, while in Section 4 their theoretical properties are presented 
Section 5 contains simulation results, Section 6 presents a real data example, and Section 7 closes the paper with a discussion  The proofs are given in the appendix to the paper 

Set up and diagonalization of Toeplitz matrices

Let Y 1 ,       , Y n i i d  ∼ N p (0 p , Σ), where Σ is a (p × p)-dimensional positive definite covariance matrix with a Toeplitz structure, that is, Σ = {σ |i−j| } p i,j=1 0  The sample size n may tend to infinity or to be a constant  The case n = 1 corresponds to a single observation of a stationary time series, and in this case the data are simply denoted by Y ∼N p (0 p , Σ) 
The dimension p is assumed to grow  The spectral density function f , corresponding to a Toeplitz covariance matrix Σ, is given by so that for f ∈ L 2 (−π, π) the inverse Fourier transform implies Hence, Σ is completely characterized by f , and the non-negativity of the spectral density function implies the positive definiteness of the covariance matrix 
Moreover, the decay of the autocovariance σ k is directly connected to the smoothness of f   Finally, the convergence rate of a Toeplitz covariance estimator and that of the corresponding spectral density estimator are directly related via Σ ≤ f ∞ := sup x∈ |f (x)|, where • denotes the spectral norm (see  
As in , we introduce a class of positive definite Toeplitz covariance matrices with Hölder continuous spectral densities  For β = γ + α > 0, where The optimal convergence rate for estimating Toeplitz covariance matrices over P β (M 0 , M 1 ) depends crucially on β  It is well known that the k-th Fourier coefficient of a function whose γ-th derivative is α-Hölder continuous decays at least with order O(k −β ) (see  
Hence, β determines the decay rate of the autocovariances σ k , which are the Fourier coefficients of the spectral density f , as k → ∞  In particular, this implies that for β ∈ (0, 1], the class P β (M 0 , M 1 ) includes Toeplitz covariance matrices corresponding to long-memory processes with bounded spectral densities, since the sequence of corresponding autocovariances is not summable 
A connection between Toeplitz covariance matrices and their spectral densities is further exploited in the following lemma  Lemma 1  Let Σ ∈ P β (M 0 , M 1 ) and let x j = (j − 1)/(p − 1), j = 1,    , p, then where δ i,j is the Kroneker delta, O(•) terms are uniform over i, j = 1,       , p and divided by √ 2 when i, j ∈ {1, p} is the Discrete Cosine Transform I (DCT-I) matrix 
The proof can be found in Appendix A 1  This result shows that the DCT-I matrix approximately diagonalizes Toeplitz covariance matrices and that the diagonalization error depends to some extent on the smoothness of the corresponding spectral density  In the spectral density literature the discrete Fourier transform (DFT) matrix
, where i is the imaginary unit, is typically employed to approximately diagonalize Toeplitz covariance matrices  Using the fact that introduced an approximation for the likelihood of a single Gaussian stationary time series (case n = 1), the so-called Whittle likelihood (1) The quantity , where F j denotes the j-th column of F , is known as the periodogram at the j-th Fourier frequency 
Note that due to periodogram symmetry, only p/2 data points I 1 ,    , I p/2 are available for estimating the mean f (2πj/p), j = 1,       , p/2 , where x denotes the largest integer strictly smaller than x  The Whittle likelihood has become a popular tool for parameter estimation of stationary time series, e g , for nonparametric and parametric spectral density estimation or for estimation of the Hurst exponent, see e g , ;  
Lemma 1 yields the following alternative version of the Whittle likelihood where W j = (D t j Y ) 2   Note that this likelihood approximation is based on twice as many data points W j as the standard Whittle likelihood  Thus, it allows for a more efficient use of the data Y to estimate the parameter of interest, such as the spectral density or the Hurst parameter 
Equations ( ) or (2) invite for the estimation of f by maximizing the (penalized) likelihood over certain linear spaces (e g , spline spaces), as suggested e g , in or   However, such an approach requires well-designed numerical methods to solve the corresponding optimization problem, since the spectral density in the second term of (1) or ( ) is in the denominator, which does not allow to obtain a closed-form expression for the estimator and often leads to numerical instabilities 
Also, the choice of the smoothing parameter becomes challenging  Therefore, we suggest an alternative approach that allows the spectral density to be estimated as a mean in an approximate Gaussian regression  Such estimators have a closed-form expression, do not require an iterative optimization algorithm and a smoothing parameter can be easily obtained with any conventional criterion 
First Hence, for W j = (D t j Y ) 2 , j = 1,       , p it follows with Lemma 1 that where Γ(a, b) denotes a gamma distribution with a shape parameter a and a scale parameter b  Note that the random variables W 1 ,       , W p are only asymptotically independent  Obviously, E(W j ) = f (πx j ) + O(1), j = 1,      
, p  To estimate f from W 1 ,       , W p , one could use a generalized nonparametric regression framework with a gamma distributed response, see e g , the classical monograph by   However, this approach requires an iterative procedure for estimation, e g , a Newton-Raphson algorithm, with a suitable choice for the smoothing parameter at each iteration step 
Deriving the L ∞ rate for the resulting estimator is also not a trivial task  Instead, we suggest to employ a variance stabilizing transform of that converts the gamma regression into an approximate Gaussian regression  In the next section we present the methodology in more detail for a general setting with n ≥ 1 

Methodology

For Y i ∼ N p (0 p , Σ), i = 1,       , n, it was shown in the previous section that with Lemma 1 the data can be transformed into gamma distributed random variables     , n, j = 1,       , p, where for each fixed i the random variable W i,j has the same distribution as W j given in (3)  Now the approach of Cai et al  ( ) is adapted to the setting n ≥ 1 
First, the transformed data points W i,j are binned, that is, fewer new variables     , T   Note that the number of observations in a bin is m = np/T   In Theorem 1 in Section 4, we show that setting T = p υ for any υ ∈ ((4 − 2 min{β, 1})/3, 1) leads to the minimax optimal rate for the spectral density estimator 
To simplify the notation, m is handled as an integer (otherwise, one can discard several observations in the last bin)  Next, applying the variance stabilizing transform (VST) ∼ where H(y) = {φ(m/2) + log (2y/m)} / √ 2 and φ is the digamma function (see   Now, the scaled and shifted log-spectral density H(f ) can be estimated with a periodic smoothing spline
where h > 0 denotes a smoothing parameter, q ∈ N is the penalty order and S per (2q − 1) a space of periodic splines of degree 2q − 1  The smoothing parameter h can be chosen either with generalized cross-validation (GCV) as derived in or with the restricted maximum likelihood, see   Once an estimator H(f ) is obtained, application of the inverse transform function H −1 (y) = m exp √ 2y − φ (m/2) /2 yields the spectral density estimator f = H −1 H(f )  
Finally, using the inverse Fourier transform leads to the fol- The precision matrix Ω is estimated by the inverse Fourier transform of the reciprocal of the spectral density estimator, i e , Ω = (ω |i−j| ) p i,j=1 with ωk = The estimation procedure for Σ and Ω can be summarised as follows  1  Data Transformation:
where D is the (p × p)-dimensional DCT-I matrix as given in Lemma 1 and D j is its j-th column  2  Binning: Set T = p υ for any υ ∈ ((4 − 2 min{β, 1})/3, 1) and calculate W i,j , k = 1,       , T 

VST:

where k are asymptotically i i d  Gaussian variables  Inverse VST: Estimate the spectral density f with f = H −1 H(f ) , where Note that Σ and Ω are positive definite matrices by construction, since their spectral density functions f and f −1 are non-negative, respectively  Unlike the banding and tapering estimators, the autocovariance estimators σk are controlled by a single smoothing parameter h, which can be estimated fully data-driven with several available automatic methods, which are numerically efficient and well-studied 
In addition, one can also use methods for adaptive mean estimation, see e g , , which in turn leads to adaptive Toeplitz covariance matrix estimation  All inferential procedures developed in the Gaussian regression context can also be adopted accordingly 

Theoretical Properties

In this section, we study the asymptotic properties of the estimators f , Σ and Ω  The results are established under the asymptotic scenario where p → ∞ and p/n → c ∈ (0, ∞], that is, the dimension p grows, while the sample size n either remains fixed or also grows but not faster than p  This corresponds to the asymptotic scenario when the sample covariance matrix is inconsistent 
Let f be the spectral density estimator defined in Section 3, i e , f = m exp{ √ 2 H(f ) − φ(m/2)}/2, where H(f ) is given in (4), m = np/T and φ is the digamma function  Furthermore, let Σ be the Toeplitz covariance matrix estimator and Ω the corresponding precision matrix defined in equations ( ) and (6), respectively 
The following theorem shows that both Σ and Ω attain the minimax optimal rate of convergence over the class and hT → ∞, then with T = p υ for any υ ∈ ((4 − 2 min{β, 1})/3, 1) and q = max{1, γ}, the spectral density estimator f , the corresponding covariance matrix estimator Σ and the precision matrix estimator Ω satisfy sup
For h {log(np)/(np)}   The proof of Theorem 1 can be found in the Appendix A 3 and is the main result of our work  The most important part of this proof is the derivation of the convergence rate for the spectral density estimator f under the L ∞ norm  In the original work, established an L 2 rate for a wavelet nonparametric mean estimator in a gamma regression where the data are assumed to be independent 
In our work, the spectral density estimator f is based on the gamma distributed data W i,1 ,       , W i,p , which are only asymptotically independent  Moreover, the mean of these data is not exactly f (πx 1 ),       , f (πx p ), but is corrupted by the diagonalization error given in Lemma 1  This error adds to the error that arises via binning and VST and that describes the deviation from a Gaussian distribution, as derived in  
Finally, we need to obtain an L ∞ rather than an L 2 rate for our spectral density estimator  Overall, the proof requires different tools than those used in   To get the L ∞ rate for f , we first derive that for the periodic smoothing spline estimator H(f ) of the log-spectral density  To do so, we use a closed-form expression of its effective kernel obtained in , thereby carefully treating various (dependent) errors that describe deviations from a Gaussian nonparametric regression with independent errors and mean f (πx i ) 
Note also that although the periodic smoothing spline estimator is obtained on T binned points, the rate is given in terms of the vector dimension p  Then, using the Cauchy-Schwarz inequality and a mean value argument, this rate is translated into the L ∞ rate for the spectral density estimator f   To obtain the rate for the Toeplitz covariance matrix estimator is enough to note that

Simulation Study

In this section, we compare the performance of the proposed Toeplitz covariance estimator, denoted as VST-DCT, with the tapering estimator of and with the sample covariance matrix  We consider Gaussian vectors Y 1 ,    , (3) such that the corresponding spectral density is Lipschitz continuous but not differentiable: f (x) = 1 44{| sin(x + 0 5π)| 1 7 + 0 45} 
In particular, var(Y i ) = 1 44 in all three examples  Figure shows the spectral densities and the corresponding autocorrelation functions for the three examples  A Monte Carlo simulation with 100 iterations is performed using R (version 4 1 2, seed 42)  For our VST-DCT estimator, we use a cubic periodic spline, i e , q = 2 is set in (4) 
The binning parameters are set to T = 500 bins with m = 10 points for (A) and T = 500 bins with m = 100 points for both (B) and (C)  To select the regularisation parameter for our estimator, we implemented the restricted maximum likelihood (ML) method, generalized cross validation (GCV) and the corresponding oracle versions, i e , as if Σ were known The tapering parameter
where T ap k (Σ ν 2 ) is the tapering estimator of with parameter k  If n = 1, that is, under scenario (A), suggest to split the time series Y into l non-overlapping subseries of length p/l and then proceed as before to select the tuning parameter k  To the best of our knowledge, there is no data-driven method for selecting this parameter l 
Using the true covariance matrix Σ, we selected l = 30 subseries for the example 1 and l = 15 subseries for the exam-ples 2 and 3  The parameter k can then be chosen by cross-validation as above  We employ this approach under scenario (A) instead of an unavailable fully data-driven criterion and name it semi-oracle 
Finally, for all three scenarios (A), (B) and (C), the oracle tapering parameter is computed using grid search for each Monte Carlo sample as kor = arg min k=2,3,   ,p/2 T ap k ( Σ) − Σ , where Σ is the sample covariance matrix  To speed up the computation, one can replace the spectral norm by the 1 norm, as suggested by  
In Tables , the errors of the Toeplitz covariance estimators with respect to the spectral norm and the computation time for one Monte Carlo iteration are given for scenarios (A), (B) and (C), respectively  To illustrate the goodness-of-fit of the spectral density, the L 2 norm f − f 2 is also computed 
The results show that the tapering and VST-DCT estimator perform overall similar in terms of the spectral norm risk  This is not surprising as both estimators are proved to be rate-optimal  Moreover, both the tapering and VST-DCT estimators are clearly superior to the inconsistent sample Toeplitz covariance matrix 
A closer look at the numbers shows that the VST-DCT method has better constants, i e , VST-DCT estimators have somewhat smaller errors in the spectral norm than the tapering estimators across all examples, but especially under scenario (C)  The oracle estimators show similar behaviour, but are slightly less variable compared to the data-driven estimators 
In general, both the tapering and VST-DCT estimators perform best for example 1, second best for example 3 and worst for example 2, which traces back to functions complexity  In terms of computational time, both methods are similarly fast for scenarios (A) and (B)  For scenario (C), the tapering method is much slower due to the multiple high-dimensional matrix multiplications in the cross-validation method 
It is expected that for larger p the tapering estimator is much more computationally intensive than the corresponding VST-DCT estimator  (1) polynomial σ ( (  To test how robust our approach is to deviations from the Gaussian assumption, we simulated the data from gamma and uniform distributions and conducted a simulation study for the same scenarios and examples 
The results are very similar to those of the Gaussian distribution, see supplementary materials for the details 

Application to Protein Dynamics

We revisit the data analysis of protein dynamics performed in Krivobokova et al  (2012) and   We consider data generated by the molecular dynamics (MD) simulations for the yeast aquaporin (Aqy1) -the gated water channel of the yeast Pichi pastoris  MD simulations are an established tool for studying biological systems at the atomic level on timescales of nano-to microseconds 
The data are given as Euclidean coordinates of all 783 atoms of Aqy1 observed in a 100 nanosecond time frame, split into 20 000 equidistant observations  Additionally, the diameter of the channel y t at time t is given, measured by the distance between two centers of mass of certain residues of the protein 
The aim of the analysis is to identify the collective motions of the atoms responsible for the channel opening  In order to model the response variable y t , which is a distance, based on the motions of the protein atoms, we chose to represent the protein structure by distances between atoms and certain fixed base points instead of Euclidean coordinates 
That is, we calculated where A t,i ∈ R 3 , i = 1,       , 783 denotes the i-th atom of the protein at time t, B j ∈ R 3 , j = 1, 2, 3, 4, is the j-th base point and d(•, •) is the Euclidean distance  Figure shows the diameter y t and the distance between the first atom and the first center of mass  It can therefore be concluded that a linear model Y = Xβ + holds, where
  This linear model has two specific features which are intrinsic to the problem: first, the observations are not independent over time and second, X t is high-dimensional at each t and only few columns of X are relevant for Y   have shown that the partial least squares (PLS) algorithm performs exceptionally well on this type of data, leading to a small-dimensional and robust representation of proteins, which is able to identify the atomic dynamics relevant for Y  
Singer et al  ( ) studied the convergence rates of the PLS algorithm for dependent observations and showed that decorrelating the data before running the PLS algorithm improves its performance  Since Y is a linear combination of columns of X, it can be assumed that Y and all columns of X have the same correlation structure 
Hence, it is sufficient to estimate Σ = cov(Y ) to decorrelate the data for the PLS algorithm, i e , Σ −1/2 Y = Σ −1/2 Xβ + Σ −1/2 results in a standard linear regression with independent errors  Our goal now is to estimate Σ and compare the performance of the PLS algorithm on original and decorrelated data 
For this purpose, we divided the data set into a training and a test set (each with p = 10 000 observations)  First, we tested whether the data are stationary  The augmented Dickey-Fuller test confirmed stationarity for Y with a p-value< 0 01  The Hurst exponent of Y is 0 85, indicating moderate long-range dependence supported by a rather slow decay of the sample autocovariances (see grey line in the left plot of Figure ) 
Therefore, we set q = 1 for the VST-DCT estimator to match the low smoothness of the corresponding spectral density  Moreover, the smoothing parameter is selected with the restricted maximum likelihood method and T = 550 bins are used  Obviously, the performance of the PLS algorithm on the decorrelated data is significantly better for the small number of components 
In particular, with just one PLS component, the correlation between the true opening diameter on the test set and its prediction that takes into account the dependence in the data is already 0 54, while it is close to zero for PLS that ignores the dependence in the data  showed that the estimator of β based on one PLS component is exactly the ensemble-weighted maximally correlated mode (ewMCM), which is defined as the collective mode of atoms that has the highest probability to achieve a specific alteration of the response Y  
Therefore, an accurate estimator of this quantity is crucial for the interpretation of the results and can only be achieved if the dependence in the data is taken into account  Estimating Σ with a tapered covariance estimator has two practical problems  First, since we only have a single realization of a time series Y (n = 1), there is no datadriven method for selecting the tapering parameter 
Second, the tapering estimator turned out not to be positive definite for the data at hand  To solve the second problem, we truncated the corresponding spectral density estimator ftap to a small positive value, i e , f + tap = max{ ftap , 1/ log(p)} (see   To select the tapering parameter with cross-validation, we experimented with different subseries lengths and found that the tapering estimator is very sensitive to this choice 
For example, estimating the tapered covariance matrix based on subseries of length 8/15/30 yields a correlation of 0 42/0 53/0 34 between the true diameter and the first PLS component, respectively  Altogether, our proposed estimator is fully data-driven, fast even for large sample sizes, automatically positive definite and can handle certain long-memory processes 
In contrast, the tapering estimator is not data-driven and must be manipulated to become positive definite  Our method is implemented in the R package vstdct 

Discussion

In this paper, we proposed a simple, fast, fully data-driven, automatically positive definite and minimax optimal estimator of Toeplitz covariance matrices from a large class that also includes covariance matrices of certain long-memory processes  Our estimator is derived under the assumption that the data are Gaussian 
However, simulations show that the suggested approach yields robust estimators even when the data are not normally distributed  In the context of spectral density estima- , for mixing processes (see Theorem 5 3 of Rosenblatt, 2012), as well as for non-linear processes (see   Since DFT and DCT matrices are closely related, we expect that equation (3) also holds asymptotically for these non-Gaussian time series, but consider a rigorous analysis to be beyond the scope of this paper 
In fact, our numerical experiments have even shown that if the spectral density is estimated from W j = f (πx j ) + j , that is, as if W j were Gaussian instead of gamma distributed, then the resulting spectral density estimator has almost the same L ∞ risk (and hence the corresponding covariance matrix has almost the same spectral norm) 
Of course, such an estimator would lead to a wrong inference about f (πx j ), since the growing variance of W j would be ignored  Since our approach translates Toeplitz covariance matrix estimation into a mean estimation in an approximate Gaussian nonparametric regression, all approaches developed in the context of Gaussian nonparametric regression, such as (locally)
adaptive estimation, as well as the corresponding (simultaneous) inference, can be directly applied  Bayesian tools for adaptive estimation and inference in Gaussian nonparametric regression as proposed in can also be employed 

Appendix

Throughout the appendix, we denote by c, c 1 , C, C 1 ,       etc  generic constants, that are independent of n and p  To simplify the notation, the constants are sometimes skipped and we write for less than or equal to up to constants  We embed the p-dimensional Toeplitz matrix Σ = toep(σ 0 ,       , σ p−1 ) in a (2p − 2)dimensional circulant matrix Σ = toep(σ 0 ,      
, σ p−1 , σ p−2 ,       , σ 1 )  Then, Σ = with the conjugate transpose U * , and Λ is a diagonal matrix with the k-th diagonal value for k = 1,    , p given by Furthermore, Σ = V * ΛV , where V ∈ C (2p−2)×p contains the first p columns of U   In particular, b(j, r) = b(j, 2p−r) and c(j, r) = −c(j, 2p−r) for r = p+1,      
, 2p−2  Together, we have (A 1) Some calculations show that for r = 1,       p Using the Taylor expansion of cot(x) for 0 < |x| < π one obtains for r = 1,       p where the O term does not depend on j and the hidden constant does not depend on r, p  If i = j, equations (A 1) -(A 3) imply where the O terms do not depend on j 
Since the complex exponential function is Lipschitz continuous with constant L = 1, it holds λ r = λ j + L r,j |r − j|p −1 where −1 ≤ L r,j ≤ 1 is a constant depending on r, j  Then, , it is sufficient to consider j = 1,    , p − 1  We begin with first sum  For a shorter notation, we use k := r − 1 and l := j − 1 in the following 
Then, summing the squares of the first term in (A 4) for l = 0,    , p−2 on sums of reciprocal powers  If p is even, then the residual terms are given by where φ and φ (1) denote the digamma function and its derivative  If p is odd, similar remainder terms can be derived  To see that R i (l, p) = O(p −1 ) for i = 1, 2, 3 and uniformly in l we use that asymptotically φ(x)∼ log(x)−1/(2x) and
The mixed term are both of the order p −1   Furthermore, since the harmonic sum diverges at a rate of log(p)  Finally, λ j = f (x j )+O{log(p)p −β } by the uniform approximation properties of the discrete Fourier series for Hölder continuous functions (see   All together, we have shown that (DΣD) j,j = where the O terms are uniform over j = 1,    , p 
Case i = j and |i − j| is even In this case, (DΣD) i,j = a i a j uniformly in i, j  To show that a i a j 2p−2 r=1 λ r c(i, r)c(j, r) = O(p −1 ), we proceed similarly as before  Setting k=r−1, l=j−1, m=i−1 and using that l =m and |l−m| is even, one obtains where for even p the residual terms are given by If p is odd, analogous residual terms can be derived 
Using similar techniques as before, one can show that the two residual terms and the remaining mixed and square terms vanish at a rate of the order O(p −1 ) and uniformly in i, j  Case i = j and |i − j| is odd |r − i| and |r − j| are either odd and even, or even and odd  Without loss of generality, assume that |r − i| is even 
Then, (DΣD) i,j = a i a j 2p−2 r=1 λ r b(i, r)c(j, r)  Since b(i, •) is an even function, c(j, •) is an odd function and λ r = λ 2p−r , it follows (DΣD) i,j = 0  The structure of the proof is as follows  First, we derive the L ∞ rate of the periodic smoothing spline estimator H(f )  Then, using the Cauchy-Schwarz inequality and a mean value argument, the convergence rate of the spectral density estimator f is
∞ the first claim of the theorem follows  Finally, we prove the second statement on the precision matrices  For the sake of clarity, some technical lemmas used in the proof are listed separately in A 4  hT → ∞, then with T = p υ for any υ ∈ ((4 − 2 min{1, β})/3, 1), the estimator H(f ) described in Section 3 with q = max{1, γ} satisfies
Proof : Application of the triangle inequality yields a bias-variance decomposition Set T = 2T − 2 and x k = (k − 1)/ T for k = 1,    , T   Using Lemma 4, we can write where Mirroring and renumerating ζ k , η k , k is similar as for Y * k , k = 1,    , T   Using the above representation, one can write First we reduce the supremum to a maximum over a finite number of points 
If q > 1, then W (•, x k ) is Lipschitz continuous with constant L > 0  In this case, it holds almost surely that sup ) is a piecewise linear function with knots at x j = j/ T   The factor (ζ k + ξ k ) can be considered as stochastic weights that do not affect the piecewise linear property  Thus, the supremum is attained at one of the knots x j = j/ T , j = 1,    , T , and (A 7) is also valid for q = 1 
Again with (a + b) 2 ≤ 2a 2 + 2b 2 we obtain We start with bounding   This requires a bound on 1 • ψ 2 denotes the sub-Gaussian norm  In case of a Gaussian random variable the norm equals to the variance  Thus with Lemma 2 and Lemma 4, we obtain Lemma 1 6 of ) then yields Recall that T = p υ for some fixed υ ∈ ((4 − 2 min{1, β})/3, 1) 
Using the inequality log(x) ≤ x a /a one can find constants x υ , C υ > 0 depending on υ but not on n, p such that log(2 T ) log(p) Next, we derive a bound for the second term The exponential decay property of the kernel K stated in Lemma 2 yields The first term in (A 9) can be bounded again with Lemma 1 6 of  
We use the fact that for not necessarily independent random variables X 1 ,    , X N R and R > 0 are constants  This is a consequence of Lemma 1 of which yields , it follows that N i=1 a i X i has a subGaussain distribution and the subGaussian norm is bounded by 2R( N i=1 a 2 i ) 1/2   See for further details on the subgaussian distribution 
T h   For the second inequality Lemma 2(ii) is used  Applying Lemma 1 6 of then yields To bound the second term in (A 9), we use the moment bounds for ξ k derived in Lemma 4  Then, for all integers > 1 Combining the error bounds (A 10) and (A 11) and choosing R=m −1/2 gives By assumption T = p υ and m = np (1−υ) for some fixed υ ∈ ((4 − 2 min{1, β})/3, 1) 
If is an integer such that ≥ 1/(1 − υ), then where we used log(x) ≤ x a /a with a = 1/(4 )  Consider 1/2 < β ≤ 1 and let 0 < χ < 1 be a constant  Applying log(x) ≤ x a /a twice with a = χ/(2 ) yields For any fixed υ∈((4 − 2 min{1, β})/3, 1) one can find an integer which is independent of n, p such that the right side of (A 12) holds 
Since p/n → c ∈ (0, ∞] and thus n/p = O(1) and p −1 = O(n −1 ), it follows for satisfying (A 12) that In total, choosing an integer Using the representation in Lemma 4 once more gives for each x ∈ [0, 1] The bounds on k in Lemma 4 imply Consider the case that β ≥ 1  In particular, q = γ and f (q) is α-Hölder continuous 
Since f is a periodic function with f (x) ∈ [δ, M 0 ] and H(y) ∝ φ(m/2)+ log (2y/m), it follows that {H(f )} (q) is also α-Hölder continuous  Extending g := H(f ) to the entire real line, we get Expanding g(t) in a Taylor series around x and using that h −1 K h is a kernel of order 2q, see Lemma 2(iii), it follows that for any x ∈ [0, 1]
where ξ x,t is a point between x and t  Using the fact that the kernel K h decays exponentially and that g (q) is α-Hölder continuous on [δ, M 0 ] with some constant L, the logarithm is Lipschitz continuous on a compact interval, it follows g = H(f ) is β-Hölder continuous  Expanding g to the entire line and using Lemma 2(iii) with
In a similar way as before, one obtains Note that T −β =o(h β ) as β > 1/2, T h → ∞ and h → 0 by assumption  Since the derived bounds are uniform for x ∈ [0, 1] it holds Putting the bounds A 13 and A 14 together gives If h > 0 such that h → 0 and hT → ∞, then with T = p υ for any υ ∈ ((4 − 2 min{1, β})/3, 1), the estimator f described in Section 3 with q = max{1, γ} satisfies
Proof : By the mean value theorem, it holds for some function g between H(f ) and To show that the second term on the right hand side of (A 15) is negligible we use the moment generating function of H(f ) ∞   In the next paragraph, we derive the asymptotic order of E[exp{λ H(f ) ∞ }] for n, p → ∞, where λ > 0 may depend on n, p or not 
By the exponential decay property of the kernel K stated in Lemma 2 holds First, H(f ) ∞ is bounded with the maximum over a finite number of points  Calculating the derivative of s : Since δ δx s(x) > 0 almost surely for x = x k , the extrema occur at x k , k = 1,    , T   Thus, for λ > 0 the moment generating function of H(f ) ∞ is bounded by
Let M j = ( T h) −1 T k=1 γ h (x j , x k ), which by Lemma 2 is bounded uniformly in j by some global constant M > 0  By the convexity of the exponential function we obtain √ 2 and by assumption 0 ≤ δ ≤ f ≤ M 0   Using Lemma 3, Q k can be written as a sum of m = np/T independent gamma random variables, i e 
The moment generating function of | log(X)| when X follows a Γ(a, b)-distribution is given by where Γ(a) is the gamma function and γ(a, b) is the lower incomplete gamma function  In particular, To derive the asymptotic order of E[exp{λ H(f ) ∞ }] for n, p → ∞ we first establish the asymptotic order of the ratio Γ(a + t)/Γ(a) for a → ∞ 
We distinguish the two cases where t is independent of a and where t linearly depends on a  Thus, for 0 < t < a and t independent of a, equation (A 17) implies for a → ∞ that Γ(a + t)/Γ(a) = O(a t )  Similarly, it can be seen that Γ(a − t)/Γ(a) = O(a −t )  If 0 < t < a and t linearly depends on a, i e  t = ca for some constant c ∈ (0, 1), then we get Γ(a ± t)/Γ(a) = O(a ±t exp{a}) for a → ∞ 
Hence, for a fixed λ not depending on n, p and such that 0 < λ < m/( √ 2M j ) we get for sufficiently large n, p If λ = cm such that 0 < λ < m/( √ 2M j ), then for sufficiently large n, p b∈{cδ/m,cM 0 /m} (bm/2) for some constant L > 1  Set K = min j=1,   , T 1/( √ 2M j ) which is a constant independent of n, p  Altogether, we showed that for 0 < λ < Km and n, p → ∞
Bounding the right hand side of (A 15) for some constants c 0 , c 1 > 0 and n, p → ∞ Since g lies between H(f ) and H(f ), and f almost surely pointwise  Thus, for C > f ∞ = M 0 it holds where c 1 := H(C − M 0 )  Applying Markov inequality for t = cm with c ∈ (0, K) and C = 2L 4/c + M 0 where c, K, L are the constants in gives
Together with Proposition 1 follows Using the fact that the spectral norm of a Toeplitz matrix is upper bounded by the sup norm of its spectral density we get sup According to the mean value theorem, for a function g between H(f ) and H(f ), it holds that some constant c 1 > 0 not depending on n, p  Chosing the same constant C as in section A 3 2 it follows
Noting that 1/f ∞ ≤ 1/δ and 2/m exp {φ(m/2)} ∈ [0 25, 1] for m ≥ 1, (A 18) implies for some constants c 2 , c 3 > 0 and n, p → ∞ Since the derived bounds hold for each Σ(f ) ∈ F β , we get all together sup This section states some technical lemmata needed for the proof of Theorem 1  The proofs can be found in the supplementary material 
The first lemma lists some properties of the kernel K h and its extension K h on the real line  The proof is based on   Lemma 2  Let h > 0 be the bandwith parameter depending on N   (i) There are constants 0 < C < ∞ and 0 < γ < 1 such that for all for x, t ∈ [0, 1] Lemma 3 states that the sum of the correlated gamma random variables in each bin can be rewritten as a sum of independent gamma random variables 
for i = 1,    , n and j = (k − 1)m + 1,    , km, and x j = (j − 1)/(2p − 2)  Finally, Lemma 4 gives explicit bounds for the stochastic and deterministic errors of the variance stabilizing transform  Thus, it quantifies the difference to an exact Gaussian regression setting  This result is a generalization of Theorem 1 of Cai et al 
(2010) adapted to our setting with n ≥ 1 observations and correlated observations  √ 2 can be written as where for the proof of the first statement  Furthermore, for x, t ∈ [0, 1] holds In particular, for some constants C 1 , C 2 > 0 depending on γ ∈ (0, 1) but not on h and x, it holds h (iii) See Lemma 15 of with p = 2q − 1 
It is sufficient to show the statement for n = 1 by independence of the Y i   Then, the number of points per bin is m = p/T   For simplicity, the index i is skipped in the following  First, we write Q k as a matrix-vector product and refactor it so that it corresponds to a sum of independent scaled χ 2 random variables 
In the second step, we calculate the scaling factors  Let E (km) be a diagonal matrix with ones on the (k − 1)m + 1,    , km-th entries and otherwise zero diagonal elements  Then, By Theorem 1 of for the gamma distribution it follows where Wi,j iid  ∼ Γ(1/2, 2 f (x * k )) and such that Cov( Wi,j , Wi,h ) = Cov(W i,j , W i,h ) for j = (k − 1)p/T + 1,    , kp/T and h ∈ {1,    , p} \ {(k − 1)p/T + 1,    , kp/T } 
Let θ be the maximum difference of the observations' means in each bin  Then, θ = max are defined via quantile coupling, it holds Z k = Φ −1 {F Q( Qk )} (see   Furthermore, define the uniform random variables Let ρ = Cov(Z k , Z l )  Then, the identity implies F Z,Z (x, y) − Φ(x)Φ(y) ≥ 0 for all x, y ∈ R ⇐⇒ ρ ≥ 0, (see  
Since Cov( Qk , Ql ) ≥ 0 and the ratio of two densities is non-negative, x = − 2/m, it follows that f Q(x) is monotone decreasing for x ≥ − 2/m  Furthermore, F Q(− m/2) ≤ 0 5 for all m ∈ N as f Q(x) is right-skewed  In particular, − m/2 ≤ F −1 Q (1/2) for all m ∈ N  Finally, since f Q(− 2/m) → φ(0) for m → ∞ there is a constant c > 0 not depending on m such that
The simulation study in Section 5 is performed in the same way, but with the uniform and the gamma distribution instead of the Gaussian distribution 
Context 3: Sowing the Seeds of Future Research: Data Sharing, Citation and Reuse in Archaeobotany
Reading: Sowing the Seeds of Future Research: Data Sharing, Citation and Reuse in Archaeobotany
University of Oxford, GB
Lisa is a post-doctoral research fellow at All Souls College, University of Oxford  Her publications include the co-authored volume The Rural Economy of Roman Britain (Britannia Monographs, 2017)  Her research interests are focussed on agricultural practices in the later prehistoric and Roman period and the utilisation of archaeobotanical data to investigate human-plant relationships 
The practices of data sharing, data citation and data reuse are all crucial aspects of the reproducibility of archaeological research  This article builds on the small number of studies reviewing data sharing and citation practices in archaeology, focussing on the data-rich sub-discipline of archaeobotany  Archaeobotany is a sub-discipline built on the time-intensive collection of data on archaeological plant remains, in order to investigate crop choice, crop husbandry, diet, vegetation and a wide range of other past human-plant relationships  Within archaeobotany, the level and form of data sharing is currently unknown  This article first reviews the form of data shared and the method of data sharing in 239 articles across 16 journals which present primary plant macrofossil studies  Second, it assesses data-citation in meta-analysis studies in 107 articles across 20 journals  Third, it assesses data reuse practices in archaeobotany, before exploring how these research practices can be improved to benefit the rigour and reuse of archaeobotanical research 
Keywords: Archaeobotany, Data reuse, Data sharing, Open science
How to Cite: Lodwick, L , 2019  Sowing the Seeds of Future Research: Data Sharing, Citation and Reuse in Archaeobotany  Open Quaternary, 5(1), p 7  DOI: http://doi org/10 5334/oq 62
Accepted on 29 May 2019 Submitted on 25 Mar 2019
Archaeology is a discipline built on the production and analysis of quantitative data pertaining to past human behaviour  As each archaeological deposit is a unique occurrence, ensuring that the data resulting from excavation and analysis are preserved and accessible is crucially important  Currently, there is a general perception of a low level of data sharing and reuse  Such a low level of data availability would prevent the assessment of research findings and the reuse of data in meta-analysis (Kansa & Kansa 2013; Moore & Richards 2015)  As observed across scientific disciplines, there is a major problem in the reproduction of scientific findings, commonly known as the ‘replication crisis’ (Costello et al  2013)  A range of intersecting debates contribute to this, including access to academic findings (open access), open data, access to software and access to methodologies, which can be broadly grouped as open science practices  Without these, the way that scientific findings can be verified and built upon is impaired  Questions of reproducibility have been raised in recent years in archaeology, with considerations of a range of practices which can improve the reproducibility of findings, and a recent call for the application of open science principles to archaeology (Marwick et al  2017)  Discussion has so far focussed on access to grey literature (Evans 2015), data sharing (Atici et al  2013), data citation practices (Marwick & Pilaar Birch 2018) and computational reproducibility (Marwick 2017), with a focus on lithics, zooarchaeological evidence, and archaeological site reports 
Quantitative assessments of current levels of data sharing, data citation and reuse remain limited in archaeology  The focus of evaluation has been on the uptake of large-scale digital archives for the preservation and dissemination of digital data, such as the Archaeology Data Service (ADS), utilised by developer-led and research projects, and recommended for use by many research funders in the UK (Richards 2002; Wright and Richards 2018)  Much less focus has been paid to the data-sharing practices of individuals or small-groups of university-based researchers who may be disseminating their research largely through journal articles  Recent work on the availability of data on lithics assemblages found a low level of data sharing (Marwick & Pilaar Birch 2018) and there are perceptions of low levels of data reuse (Huggett 2018; Kintigh et al  2018)  Within zooarchaeology numerous studies have explored issues of data sharing and reuse (Kansa & Kansa 2013, 2014), and the sub-discipline is seen as one of the most advanced areas of archaeology in regards to open science (Cooper & Green 2016: 273)  Beyond zooarchaeology, however, explicit discussion has remained limited 
This paper assesses data sharing and reuse practices in archaeology through the case study of archaeobotany – a long established sub-discipline within archaeology which has well-established principles of data recording  Archaeobotany is an interesting case study for data sharing in archaeology as it straddles the division of archaeology between scientific and more traditional techniques  Quantitative data on archaeological plant remains are also of interest to a range of other fields, including ecology, environmental studies, biology and earth sciences  The key issues of data sharing and data reuse (Atici et al  2013) have been touched upon in archaeobotany over the past decade within broader discussions on data quality (Van der Veen, Livarda & Hill 2007; Van der Veen, Hill & Livarda 2013)  These earlier studies focussed on the quality and availability of archaeobotanical data from developer-funded excavations in Britain and Cultural Resource Management in North America (Vanderwarker et al  2016: 156)  However, no discussion of data-sharing and reuse in academic archaeobotany occurred  A recent review of digital methods in archaeobotany is the notable exception, with discussions of the challenges and methods of data sharing (Warinner & d’Alpoim Guedes 2014) 
Currently, we have no evidence for the levels of data sharing and reuse within archaeobotany  This article provides the first quantitative assessment of 1) data publication in recent archaeobotanical journal articles 2) data citation in recent archaeobotanical meta-analysis 3) the reuse of archaeobotanical datasets, in order to assess whether practices need to change and how such changes can take place 
2  Data Publication and Re-use Practices in Archaeobotany
2 1  History of data production and publication
Archaeobotanical data falls within the category of observational data in archaeology (Marwick & Pilaar Birch 2018)  Archaeobotanical data is considered as the quantitative assessment of plant macrofossils present within a sample from a discrete archaeological context, which can include species identification, plant part, levels of identification (cf  – confer or “compares to”), and a range of quantification methods including count, minimum number of individuals, levels of abundance and weight (Popper 1988)  Archaeobotanical data is usually entered into a two-way data table organised by sample number  Alongside the counts of individual taxa, other information is also necessary to interpret archaeobotanical data, including sample volume, flot volume, charcoal volume, flot weight, level of preservation, sample number, context number, feature number, feature type and period  Beyond taxonomic identifications, a range of other types of data are increasingly gathered on individual plant macrofossils (morphometric measurements, isotopic values, aDNA) 
Archaeobotanical training places a strong emphasis on recording data on a sample-by-sample basis (Jacomet & Kreuz 1999: 138–139; Jones & Charles 2009; Pearsall 2016: 97–107)  Time-consuming methodologies utilised in the pursuit of accurate sample-level data recording include sub-sampling and splitting samples into size fractions and counting a statistically useful number of items per sample (Van der Veen & Fieller 1982)  The creation of sample-level data means analysis is often undertaken on the basis of individual samples, for instance the assessment of crop-processing stages and weed ecological evidence for crop husbandry practices  The analysis of sample level data also enables archaeobotanical finds to be integrated alongside contextual evidence from archaeological sites  Requirements for the publication of this data are in place in some archaeological guidelines, for instance current Historic England guidelines for archaeological practice in England (Campbell, Moffett & Straker 2011: 8) 
From the earliest archaeobotanical reports, such as Reid’s work at Roman Silchester, the sample from which plant remains were recovered was noted (Lodwick 2017a), but often results were reported as a list of taxa, or long catalogues of detailed botanical descriptions with seed counts, such as Knörzer’s work at Neuss (Knörzer 1970)  Early systematic archaeobotanical reports displayed data within in-text tables, for example Jones’s work at Ashville (Jones 1978) and the two-way data table has been the standard form of reporting archaeobotanical data ever since  Often data tables are presented within book chapters or appendices, but the financial, space and time constraints of book publishing are limiting  Furthermore, there is the perception that specialist data was not necessary for publication (Barker 2001)  Hence, alternative methods of the dissemination of specialist archaeological data were pursued in the later twentieth century 
From the 1980s, archaeobotanical data tables were often consigned to microfiche following a Council for British Archaeology and Department of Environment report (Moore & Richards 2015: 31), with the example of the excavation of Roman Colchester where the contents of all archaeobotanical samples were available on microfiche (Murphy 1992)  An alternative in the 2000s was providing data tables on CD Rom as seen, for instance, in the CD accompanying the study of a Roman farmstead in the Upper Thames Valley (Robinson 2007) or the One Poultry excavations in London (Hill and Rowsome 2011)  Meanwhile, the inception of the Archaeology Data Service, a digital repository for heritage data, in 1996 meant archaeological datasets were increasingly digitally archived, for instance the data from the Channel Tunnel Rail Link Project (Foreman 2018) or a recent large-scale research excavation at Silchester (University of Reading 2018)  In these cases, archaeobotanical data is available to download as a  csv file 
Whilst the data publication strategy of large excavations was shifting, the availability of data from post-excavation assessment reports has remained challenging  So-called ‘grey literature’ results from the initial evaluation stage of developer-funded investigations and accompanying post-excavation assessment often contain a semi-quantitative evaluation of archaeobotanical samples on a scale of abundance  Whilst paper reports were initially deposited with county Historic Environment Records, a process of digitisation focussing on the Roman period has meant many pdfs are now available through the ADS (Allen et al  2018), whilst born-digital reports are now deposited through OASIS (Online AccesS to the Index of archaeological investigationS), as part of the reporting process (Evans 2015), althought the extent to which specialist appendices are included is variable 
These varying ‘publication’ strategies means archaeobotanical data is often available somewhere for recent developer-funded excavations and large-scale developer-funded excavations, even if much of this data is as a printed table or  pdf file (Evans 2015; Evans and Moore 2014)  However, academic journals are typically perceived as the most high-status publication venue for archaeobotanical data, and a crucial publication venue for academics in order to comply with institutional requirements and the norms of career progression  Aside from the problem of access to pay-walled journals by those without institutional subscriptions to all journals, the publication of primary data alongside research articles faces various problems, from the outright lack of inclusion of data, to problematic curation of supplementary data and a lack of peer review of data (Costello et al  2013; Warinner and d’Alpoim Guedes 2014: 155; Whitlock, 2011)  The extent of these problems for archaeobotany is currently unknown  Given the growth in archaeobotanical data production as methodologies are introduced into many new regions and periods over the last decade, it is vital that we know whether the mass of new data being produced is made available and is being reused 
Recent important advances within archaeobotanical data sharing have focussed on the construction of the ARBODAT database, developed by Angela Kreuz at the Kommission für Archäologische Landesforschung in Hessen  The database is used by a range of researchers in Germany, the Czech Republic, France and England (Kreuz & Schäfer 2002)  Data sharing enabled by the use of this database has facilitated research on Neolithic agriculture in Austria, Bulgaria and Germany (Kreuz et al  2005), and Bronze Age agriculture in Europe (Stika and Heiss 2012)  The use of this database makes data integration between specialists easier due to the shared data structure and metadata description, but often the primary archaeobotanical data is not made publicly available 
2 2  Meta-analysis in archaeobotany
Beyond the need to preserve information, a key reason for the formal sharing of archaeobotanical data is in its reuse to facilitate subsequent research  There has been a long-standing concern within archaeobotany with the need to aggregate datasets and identify temporal and spatial patterns  The palaeobotanist Clement Reid maintained his own database of Quaternary plant records in the late nineteenth century (Reid 1899), which formed the foundation of Godwin’s Quaternary database (Godwin 1975)  Mid-twentieth century studies of prehistoric plant use compiled lists of archaeobotanical materials incorporating full references and the location of the archive (Jessen & Helbaek 1944)  The International Work Group for Palaeoethnobotany was itself founded in 1968 in part with the aim to compile archaeobotanical data, first realised through the publication of Progress in Old World Palaeoethnobotany (Van Zeist, Wasylikowa & Behre 1991), and subsequently through the publication of annual lists of new records of cultivated plants (Kroll 1997) 
To take England as an example, regional reviews produced by state heritage authorities have provided catalogues of archaeobotanical datasets in particular time periods and regions (e g  Murphy 1998)  When one archaeobotanist has undertaken the majority of study within a region, pieces of synthesis within books have provided a relatively comprehensive review, for instance in the Thames Valley, UK (Lambrick & Robinson 2009)  Over the last decade regional synthesis has occurred within several funded reviews which produced catalogues of sites with archaeobotanical data (Lodwick 2014; McKerracher 2018; Parks 2012) and a series of funded projects in France have enabled regional synthesis (Lepetz & Zech-Matterne 2017)  However, many of these reviews are not accompanied by an available underlying database, and draw upon reports which are themselves hard to access 
Through the 1990s and 2000s, a series of databases were constructed in order to collate data from sites in a particular region and facilitate synthetic research  However, these databases have all placed the role of data archiving onto later projects specifically funded to collate data, rather than sourcing datasets at the time of publication  Such a model is unsustainable, and is unlikely to result in all available datasets being compiled  The Archaeobotanical Computer Database (ABCD), published in 1996 in the first issue of Internet Archaeology, contained much of the archaeobotanical data from Britain available at the time of publication, largely at the level of individual samples  The database was compiled between 1989 and 1994 and is still accessible through the accompanying online journal publication (Tomlinson & Hall 1996)  The ABCD made major contributions to recent reviews of the Roman and Medieval periods (Van der Veen, Livarda & Hill 2008; Van der Veen, Hill & Livarda 2013)  However, the database could only be centrally updated, with the online resource remaining a static version, lacking much of the new data produced subsequent to the implementation of PPG16 in 1990  The ADEMNES database, created through a research project undertaken at the Universities of Freiburg and Tübingen, contains data from 533 eastern Mediterranean and Near Eastern sites (Riehl & Kümmel 2005)  Kroll has maintained the Archaeobotanical Literature Database to accompany the Vegetation History and Archaeobotany articles (Kroll 2005) now accessible as a database (Kirleis & Schmültz 2018)  Numerous other databases have collated archaeobotanical studies, including the COMPAG project (Fuller et al  2015), the Cultural Evolution of Neolithic Europe project (Colledge 2016), RADAR in the Netherlands (van Haaster and Brinkkemper 1995), BRAIN Botanical Records of Archaeobotany Italian Network (Mercuri et al  2015) and CZAD – Archaeobotanical database of Czech Republic (CZAD 2019) 
The majority of databases have a restricted regional coverage, whilst research-project driven period-specific databases provide overlapping content  Whilst there are a wide range of archaeobotanical databases available, few contain primary datasets (other than the ABCD) which can be downloaded as  csv files  Data which is most commonly available are bibliographic references per site, with some indications of mode of preservation, quantity of archaeobotanical data, and sometimes taxa present  The databases do not inter-relate to each other, and function primarily as bibliographic sources enabling researchers to find comparative sites or to identify published datasets which need to be re-tabulated prior to meta-analysis  The IWGP website curates a list of resources, but otherwise the resources are often disseminated through the archaeobotany jiscmail list 
Beyond the aim of cataloguing archaeobotanical data within a region and period, meta-analysis is often used in archaeobotany to identify spatial and chronological trends in a range of past human activities, for instance crop choice, crop husbandry practices, plant food consumption, the trade in luxury foods or the use of plants in ritual  Meta-analysis can be undertaken on the basis of simple presence/absence data per site, but in order for such analysis to be rigorous and comparable, sample-level data must be utilised  For instance, sample-level data is required for meta-studies, in order to identify high-quality samples of unmixed crops for weed ecology analysis (Bogaard 2004), to assess the importance of context in the evaluation of wild plant foods (Wallace et al  2019), or to use volumetric measurements as a proxy for scale (Lodwick 2017b)  The reuse of archaeobotanical data also extends to include datasets used as “controls” in commonly used forms of statistical analysis, for instance Jones’s weed data from Amorgos, Greece, which is utilised as a control group in discriminant analysis of crop-processing stage (Jones 1984), and ethnographic observations of crop items in different crop-processing stages (Jones 1990) 
2 3  Open data principles and solutions
Debates over issues of data publication and meta-analysis have been on-going across scientific disciplines over the last decade (Editors 2009), and have been summarised within principles of open science, as recently set out in relation to archaeology (Marwick et al  2017)  Open Data is one of the three core principles for promoting transparency in social science (Miguel et al  2014)  The FAIR principles, developed by representatives from academia, industry, funding agencies, industry and publishers, provide four principles which data sharing should meet for use by both humans and machines – Findability, Accessibility, Interoperability, and Reusability (Wilkinson et al  2016)  A recent report assessing the adoption and impact of FAIR principles across academia in the UK included archaeology as a case study (Allen and Hartland 2018: 46)  It reported how the ADS was often used to archive data, but that “The journal itself provides the “story” about the data, the layer that describes what the data is, how it was collected and what the author thinks it means ” The report also raises the problem that smaller projects may not have the funding to utilise the ADS, meaning that other repositories are utilised  Increasingly, archaeological data is made available through a wide range of data repositories (OSF, Mendeley Data, Zenodo, Open Context), university data repositories (e g  ORA-Data), or social networking sites for academics (Academia edu, ResearchGate)  More widely in archaeology, some have observed that archaeological data is rarely published (Kintigh et al  2014), and recent reviews have reported low levels of data sharing (Huggett 2018; Marwick & Pilaar Birch 2018)  A closely related issue is that of data reuse  Responsible reuse of primary data encourages the sharing of primary data (Atici et al  2013), but levels of data reuse in archaeology are thought to remain low (Huggett 2018)  Principles for responsible data citation in archaeology have recently been developed summarising how datasets should be cited (Marwick & Pilaar Birch 2018) 
In order to assess the current status of data sharing, citation and data re-use in archaeobotany, a review was undertaken of the publication of primary data and the publication of meta-analysis in major archaeological journals over the last ten years, building on recent pilot studies within archaeology (Marwick & Pilaar Birch 2018)  The review of academic journals provided a contrast to recent assessments of archaeobotanical data deriving from developer-funded archaeology (Lodwick 2017c; Van der Veen, Hill & Livarda 2013)  Journal articles have been selected as the focus of this study as the provision of online supplementary materials in the majority of journals and the ability to insert hyperlinks to persistent identifiers (eg a DOI) to link to datasets available elsewhere should not limit the publication of data and references  Much archaeobotanical data is also published elsewhere, especially from projects not based in the university sector, that is commercial or community archaeology in the UK  Archaeobotanical datasets emanating from this research are more commonly published through monographs, county journal articles, and unpublished (or grey literature) reports, but these are beyond the scope of the current review 
All journal articles were included which represent the principle reporting of a new archaeobotanical assemblage  The selected journals fall within three groups  First, what is considered the specialist archaeobotanical journal (Vegetation History and Archaeobotany (VHA))  Second, archaeological science journals (Archaeological and Anthropological Sciences, Environmental Archaeology, The Holocene, Journal of Archaeological Science (JAS), Journal of Archaeological Science: Reports (JASR), Journal of Ethnobiology, Quaternary International, Journal of Wetland Archaeology), which can be considered as specialist sub-disciplinary journals which should be maintaining data-quality  Third, general archaeology journals (Antiquity, Journal of Field Archaeology, Oxford Journal of Archaeology, Journal of Anthropological Archaeology, Journal of World Prehistory)  Finally, the broader cross-disciplinary journals PLoS One and Proceedings of the National Academy of Sciences (PNAS) were included  Published articles from the past ten years (2009–2018) have been analysed in order to assess the availability of plant macrofossil data  This ten-year period brackets the period where most archaeological journals have moved online and adopted supplementary materials 
Data citation in synthetic studies has been assessed in the same range of publications  The extent of data reuse ranges from the analysis of whole sample data to the presence/absence of individual crops  The location of a data citation has been assessed in the same range of publications, with the addition of journals where occasional research incorporating archaeobotanical data is featured (Britannia, Journal of Archaeological Research, Ethnobiology Letters, Medieval Archaeology, Proceedings of the Prehistoric Society, World Archaeology)  The underlying dataset for the analysis is available in Lodwick 2019 
4 1  Primary data sharing
Here, the location of primary archaeobotanical data, that is sample level counts of macroscopic plant remains, was assessed for 239 journal articles across 16 journals (Lodwick 2019 Table 1)  Figure 1 shows the results grouped by journal  Overall, only 56% of articles shared their primary data  In, Antiquity, JAS, JASR, PLOS One, Quaternary International and VHA, the highest proportion of publications did not include their primary data, that is to say that the sample-by-sample counts of plant macrofossils was not available  This level of data is comparable to the findings of other pilot studies in archaeology  Marwick and Pilaar Birch found a data sharing rate of 53% from 48 articles published in Journal of Archaeological Science in Feb – May 2017 (Marwick & Pilaar Birch 2018: 7), and confirm previous assertions that data is often withheld in archaeology (Kansa 2012: 499)  This is better than some disciplines, with a 9% data sharing rate on publication found across high impact journal science publications (n = 500) (Alsheikh-Ali et al  2011) and 13% in biology, chemistry, mathematics and physics (n = 4370) (Womack 2015), yet still indicates that nearly half of articles did not include primary data  Primary archaeobotanical data is more likely to be shared in archaeobotanical and archaeological science journals than general archaeology journals  However, within the primary archaeobotanical journal, VHA, 51% of articles do not include their primary data (Figure 1) 
Chart showing the location of primary archaeobotanical data by journal in primary archaeobotanical data publications 
Where primary data was not shared, the data which was available ranged from summary statistics, typically counts or frequencies, reported either by site, site phase, or feature group  Figure 2 summarises these results by year, showing that there is a gradient within articles not sharing their full ‘raw’ data, from those only provided sample counts on one aspect of the archaeobotanical assemblage, to those only presenting data graphically or within discussion  Beyond full data, the most common form of data shared is either summary counts per site or summary counts per feature or phase  Whilst this data does enable some level of reuse, the results of any sample-level data analysis presented within an article cannot be verified, and the data cannot be reused for crop-processing or weed ecology analysis which requires sample level data  Furthermore, such data would have been collected on a sample-by-sample basis, but this information is lost from the resulting publication 
Chart showing the form of archaeobotanical data shared by year in primary archaeobotanical data publications 
The forms in which data are made available vary across journals  The sharing of primary data within an article remains the most common data sharing form in archaeobotany (Figure 1)  Data tables in text require manual handling to extract data, in journals such as VHA, whilst in other journals in-text tables can be downloaded as  csv files  These however would not be citable as a separate dataset  Supplementary datasets are the third most common form of data sharing  Indeed, the use of electronic supplementary material has been advocated recently for by some journals, such as the Journal of Archaeological Science (Torrence, Martinón-Torres & Rehren 2015)  Microsoft Excel spreadsheets are the most common form of supplementary data, followed by  pdfs and then word documents (Figure 1)  Both  xlsx and  docx are proprietary file formats, and not recommended for long term archiving or open science principles  There is no indication of improvement over the last decade in the form of data sharing  In 2018, 50% of articles did not share their primary data, and where the data was shared, it was in proprietary forms ( docx,  xlsx) or those that do not easily facilitate data reuse ( pdf) (Figure 3) 
Chart showing the location of archaeobotanical data from 2009–2018 in primary archaeobotanical data publications 
Just one of the articles included in this review incorporated a dataset archived in a repository (Farahani 2018), in contrast to the substantial growth in data repositories across academic disciplines (Marcial & Hemminger 2010)  Other examples provide the underlying data for monograph publications, such as that of the archaeobotanical data from Gordion, Turkey (Marston 2017a, 2017b), Silchester, UK (Lodwick 2018; University of Reading 2018) and Vaihingen, Germany (Bogaard 2011a; Bogaard, 2011b) 
Several of the journals that have been assessed have research data policies  In the case of Vegetation History and Archaeobotany, sufficient papers have been surveyed to assess the impact of the research data policy on the availability of data  Figure 4 show the proportion of data sharing formats through time just for VHA (note the small sample size)  The introduction of a research data policy in 2016 encouraging data sharing in repositories has not resulted in any datasets being shared in that format  Of the 10 articles published in PLOS One after the introduction of a clear research data policy in 2014, 4 did not contain primary data  However, elsewhere, journals with no research data policy, such as Antiquity, has one of the lower levels of data sharing (Figure 1) 
Chart showing the location of primary archaeobotanical data in Vegetation History and Archaeobotany 
There are various reasons for why a primary dataset may be lacking  The option of providing supplementary datasets has been available in many of the journals here since before the start of the surveyed period (e g  Vegetation History and Archaeobotany in 2004), and so cannot be a reason for the absence of data publication in this journal while it may be a reason in other journals  Reasons suggested for a lack of data sharing within archaeology include technological limitations, and resistance amongst some archaeologists to making their data available due to cautions of exposing data to scrutiny, lost opportunities of analysis before others use it and loss of ‘capital’ of data (Moore & Richards 2015: 34–35)  Furthermore, control over how data tables is presented (taxa ordering, summary data presented) may also contribute to the preferential publishing of data within journal articles  Another factor to consider is the emphasis on the creation of new data through archaeological research (Huvila 2016)  The creation of a new archaeobotanical dataset through primary analysis is a key form of training in archaeobotany, and the perception of the value of the reuse of other previously published archaeobotanical journals may be low, hence not encouraging the sharing of well-documented datasets  Excellent exams of data reuse have resulted in influential studies (Bogaard 2004; Riehl 2008; Wallace et al  2019), and would hopefully encourage further data sharing in the future 
Given that there are numerous examples of meta-analysis which do take place in archaeobotany, it seems likely that the prevalent form of data sharing is through informal data sharing between individual specialists  However, this does not improve access to data in the long term, and is inefficient and time consuming, with large potential for data errors (Kansa & Kansa 2013), and relies on personal networks, which are likely to exclude some researchers  The absence of primary data in many archaeobotanical publications thus inhibits the verification of patterns observed within a dataset, and strongly limits the re-use potential of a dataset 
4 2  Data citation
One of the common arguments for increasing data sharing is an associated increase in the citation of the articles which have data available  Here, the data citation practices of meta-analyses of plant macrofossil data undertaken over the last decade have been reviewed  20 journals were consulted, including a wider range of period-specific journals, and 107 articles were assessed (Lodwick 2019 Table 2)  Data citation was assessed as ‘in text’ or ‘in table’ to refer to when the citation and the bibliographic reference were within the article, as ‘in supplementary data’ when the citation and reference were within the supplementary materials, and as ‘no citation’ when no citation and reference was provided 
21% of articles (n = 22) did not contain any citations to the underlying studies  16% (n = 17) contained citations within supplementary data files  50% of articles (n = 53) contained a citation within a table within the main article, and 14% (n = 15) contained citations within the main text  For the 21% of articles without data citations, the results of these studies could not be reproduced without consulting individual authors  The papers supplying the underlying data also received no credit for producing these datasets  Where articles contain citations within the main article (in text or table), full credit is provided to the underlying studies, a citation link is created through systems such as google scholar, and the study can be easily built upon in the future  Where the citation is provided within supplementary data, the original studies do receive attribution, but are not linked to so easily 
Through time, there is a steady decrease in the proportion of studies without citations to the underlying data, whereby of the 17 meta-analysis articles published in 2018, only one had no data citations  In comparison, in 2009, 3 out of 8 meta-analysis articles contained no data citation (Figure 6)  Overall this is a more positive outlook on the reuse of published data, but the consistent presence of articles lacking data citation indicates that improvements are needed  Reasons for a lack of data citation may include restrictions on word counts imposed by journals, a lack of technical knowledge in making large databases available, or the wish to hold on to a dataset to optimise usage  Considering the type of journal (Figure 5), levels of data citation are worse in general archaeology journals, with sub-disciplinary journals showing slightly better levels of data citation  In particular VHA has a lack of consistency in where data citations are located 
Chart showing the location of data citations in meta-analysis journal articles by journal type 
Chart showing the location of data citations in meta-analysis journal articles from 2009–2018 
4 3  Reuse of archived archaeobotanical datasets
The majority of data citations assessed in the previous section are to articles or book chapters rather than data-sets  The ADS currently hosts 66 data archives which have been tagged as containing plant macro data, deriving mainly from developer-funded excavations but also some research excavations  However, in some of these the plant macro data is contained within a pdf  As, the archiving of archaeobotanical datasets in data repositories is still at an early stage, the reuse of these datasets is assessed here on a case-by-case basis  The archaeobotanical dataset from the Neolithic site of Vaihingen, Germany (Bogaard 2011b) has not been cited on google scholar  Metrics are provided through the ADS, showing this dataset has been downloaded 56 times with 477 individual visits (as of 25/2/19)  The archaeobotanical dataset from Gordion by Marston has no citations on Google Scholar (Marston 2017b), neither does the Giza botanical database (Malleson & Miracle 2018), but these are both very recently archived datasets  In contrast, the Roman Rural Settlement Project dataset, which includes site-level archaeobotanical data, has received greater levels of use, with 12 citations in Google Scholar, over 40,000 file downloads, and over 35,000 visits (Allen et al  2018) and the archaeobotanical computer database (Tomlinson & Hall 1996) has been cited 44 times, and is the major dataset underpinning other highly-cited studies (Van der Veen, Livarda & Hill 2008; Van der Veen, Hill & Livarda 2013)  Whilst there is clearly precedence for the reuse of archaeobotanical databases, current data citation practices within archaeobotany do not yet appear to be formally citing individual datasets, meaning an assessment of the reuse of archived archaeobotanical datasets is challenging 
5  Steps Forward
This review of data sharing, citation, and reuse practices in archaeobotany has found medium levels of data sharing, good levels of data citation, but so far limited levels of reuse of archived data sets  This picture is similar across archaeology, in part attributed to the status of archaeology as a small-science, where data-sharing takes place ad-hoc (Marwick & Pilaar Birch 2018)  Here, recommendations are discussed for improving these data practices within archaeobotany, of applicability more widely in archaeology 
Clearly an important step is improving the sharing of plant macrofossil data  Given the reasonable small size of most archaeobotanical datasets (a  csv file < 1mb), and a lack of ethical conflicts, there seems to be few reasons why the majority of archaeobotanical data couldn’t be shared  In the case of developer-funded derived data, issues of commercial confidentiality could limit the sharing of data  A key stage is establishing why levels of data sharing are not higher  Issues within archaeobotany may include the conflict between having to publish results within excavation monographs, which may take some time to be published, and have limited visibility due to high purchase costs and no digital access, and the need to publish journal articles for career progression within academia  The production of an archaeobotanical dataset is very time-consuming, and interim publication on notable aspects of an assemblage may be considered as a necessary publication strategy  More broadly, one important aspect is issues of equity in access to digital archiving resources (Wright & Richards 2018), such as differential access to funds, training and knowledge  A recent study in Sweden found that we need to know concerns, needs, and wishes of archaeologists in order to improve preservation of archaeological data (Huvila 2016), especially when control of ones data may be linked to perceptions of job security  In order to make improvements in data sharing and reuse across archaeology, we need improved training in data sharing and the reuse of data in higher education (Touchon & McCoy 2016; Cook et al  2018), improved training in data management (Faniel et al  2018), and crucially, the necessary software skills to make the reuse of archived datasets attainable (Kansa & Kansa 2014: 91)  Examples of good practice in archaeobotany are the Vaihingen and Gordion datasets which demonstrate how datasets can be archived in data repositories to accompany a monograph (Bogaard 2011b; Marston 2017b), whilst Farahani (2018) provides an excellent example of a journal article, where the primary data is supplied as a  csv in a cited data repository along with the R script for the analysis 
In tandem with the need to encourage authors to share their data, is the need for journals to create and implement research data policies  Given the existence of research data policies in many of the journals included here, this reflects other findings of the poor enforcement of data policies by journals (Marwick & Pilaar Birch 2018), supporting arguments that journals should not be relied upon to make data accessible, and data should instead by deposited in digital repositries  In order to implement change in data sharing, there is a role to play for learned societies and academic organisation in lobbying funding bodies, prioritising data sharing in research projects  A key step is through journal editorial boards, and the enforcement of any pre-existing research data policies (Nosek et al  2015)  Revi



ground_truth: $\int f'(u) \psi^2 \le \int | \nabla \psi|^2, \forall \psi \in C_c^2$.

prompt_front: Now, answer the following question based on the text below, only give me the answer and do not output any other words.

Question: What is the advantage of decorrelating the data before running the PLS algorithm?

Read the following text and answer briefly.

Context 4: Weep Not, Child is a 1964 novel by Kenyan author Ngũgĩ wa Thiong'o  It was his first novel, published in 1964 under the name James Ngugi  It was among the African Writers Series  It was the first English language|English novel to be published by an East African  Thiong'o's works deal with the relationship between Africans and white settlers in colonial Kenya, and are heavily critical of colonial rule  Specifically, Weep Not, Child deals with the Mau Mau Uprising, and "the bewildering dispossession of an entire people from their ancestral land " Ngũgĩ wrote the novel while he was a student at Makerere University 

The book is divided into two parts and eighteen chapters  Part one deals mostly with the education of Njoroge, while part two deals with the rising Mau Mau movement 

Plot summary

Njoroge, a little boy, is urged to attend school by his mother  He is the first one of his family able to go to school  His family lives on the land of Jacobo, an African made rich by his dealings with white settlers, namely Mr  Howlands, the most powerful land owner in the area  Njoroge's brother Kamau works as an apprentice to a carpenter, while Boro, the eldest living son, is troubled by his experiences while in forced service during World War II, including witnessing the death of his elder brother  Ngotho, Njoroge's father and a respected man in the surrounding area, tends Mr  Howlands' crops, but is motivated by his passion to preserve his ancestral land, rather than for any compensation or loyalty 

One day, black workers call for a strike to obtain higher wages  Ngotho is ambivalent about participating in the strike because he fears he will lose his job  However, he decides to go to the gathering, even though his two wives do not agree  At the demonstration, there are calls for higher wages  Suddenly, the white police inspector brings Jacobo to the gathering to pacify the native people  Jacobo tries to put an end to the strike  Ngotho attacks Jacobo, and the result is a riot where two people are killed  Jacobo survives and swears revenge  Ngotho loses his job and Njoroge’s family is forced to move  Njoroge’s brothers fund his education and seem to lose respect for their father 

Mwihaki, Jacobo's daughter and Njoroge's best friend, enters a girls' only boarding school, leaving Njoroge relatively alone  He reflects upon her leaving, and realizes that he was embarrassed by his father's actions towards Jacobo  For this reason, Njoroge is not upset by her exit and their separation  Njoroge switches to another school 

For a time, everyone's attention is focused on the upcoming trial of Jomo Kenyatta – a revered leader of the movement  Many blacks think that he is going to bring forth Kenya’s independence  But Jomo loses the trial and is imprisoned  This results in further protests and greater suppression of the black population 

Jacobo and a white landowner, Mr  Howlands, fight against the rising activities of the Mau Mau, an organization striving for Kenyan economic, political, and cultural independence  Jacobo accuses Ngotho of being the leader of the Mau Mau and tries to imprison the whole family  Meanwhile, the situation in the country is deteriorating  Six black men are taken out of their houses and executed in the woods 

One day Njoroge meets Mwihaki again, who has returned from boarding school  Although Njoroge had planned to avoid her due to the conflict between their fathers, their friendship is unaffected  Njoroge passes an important exam that allows him to advance to High School  His village is proud of him, and collects money to pay Njoroge's High School tuition 

Several months later, Jacobo is murdered in his office by a member of the Mau Mau  Mr  Howlands has Njoroge removed from school for questioning  Both father and son are brutally beaten before release and Ngotho is left barely alive



ground_truth: Decorrelating the data before running the PLS algorithm improves the performance of the algorithm.

prompt_front: Now, answer the following question based on the text below, only give me the answer and do not output any other words.

Question: What are some reasons for the lack of data sharing in archaeobotany?

Read the following text and answer briefly.

Context 4:  Although there doesn't seem to be a connection between Njoroge's family and the murder, it is eventually revealed that Njoroge's brothers are behind the assassination, and that Boro is the real leader of the Mau Mau  Ngotho soon dies from his injuries and Njoroge finds out that his father was protecting his brothers  Kamau has been imprisoned for life  Only Njoroge and his two mothers remain free, and Njoroge is left as the sole provider of his two mothers  Njoroge fears that he cannot make ends meet; he gives up hope of continuing in school and loses faith in God 

Njoroge asks Mwihaki's for support, but she is angry because of her father’s death  When he finally pledges his love to her, she refuses to leave with him, realizing her obligation to Kenya and her mother  Njoroge decides to leave town and makes an attempt at suicide; however, he fails when his mothers find him before he is able to hang himself  The novel closes with Njoroge feeling hopeless, and ashamed of cowardice 

Characters in Weep Not, Child
 Njoroge: the main character of the book whose main goal throughout the book is to become as educated as possible 
 Ngotho: Njoroge's father  He works for Mr Howlands and is respected by him until he attacks Jacobo at a workers strike  He is fired and the family is forced to move to another section of the country  Over the course of the book his position as the central power of the family weakened, to the point where his self-realization that he has spent his whole life waiting for the prophecy (that proclaims the blacks will be returned their land) to come true rather than fighting for Kenyan independence, leads to his depression 
 Nyokabi and Njeri: the two wives of Ngotho  Njeri is Ngotho's first wife, and mother of Boro, Kamau, and Kori  Nyokabi is his second wife, and the mother of Njoroge and Mwangi 
 Njoroge has four brothers: Boro, Kamau, Kori and Mwangi (who is Njoroge's only full brother, who died in World War II) 
 Boro: Son of Njeri who fights for the Allies in World War II  Upon returning his anger against the colonial government is compounded by their confiscation of the his land  Boro's anger and position as eldest son leads him to question and ridicule Ngotho, which eventually defeats their father's will (upon realizing his life was wasted waiting and not acting)  It is eventually revealed that Boro is the leader of the Mau Mau (earlier alluded to as "entering politics") and murders Mr Howlands  He is caught by police immediately after and is scheduled to be executed by the book's end  It is highly likely that it is also Boro who kills Jacobo 
 Mwihaki: Njoroge's best friend (and later develops into his love interest)  Daughter of Jacobo  When it is revealed that his family killed Jacobo (most likely Boro), Mwihaki distances herself from Njoroge, asking for time to mourn her father and care for her mother 
 Jacobo: Mwihaki's father and an important landowner  Chief of the village 
 Mr  Howlands: A white settler who emigrated to colonial Kenya and now owns a farm made up of land that originally belonged to Ngotho's ancestors  Has three children: Peter who died in World War II before the book's beginning, a daughter who becomes a missionary, and Stephen who met Njoroge while the two were in high school 

Themes and motifs
Weep Not, Child integrates Gikuyu mythology and the ideology of nationalism that serves as catalyst for much of the novel's action  The novel explores the negative aspects of colonial rule over Kenya  Njoroge's aspiration to attend university is frustrated by both the violence of the Mau Mau rebels and the violent response of the colonial government  This disappointment leads to his alienation from his family and ultimately his suicide attempt 

The novel also ponders the role of saviours and salvation  The author notes in his The River Between: "Salvation shall come from the hills  From the blood that flows in me, I say from the same tree, a son shall rise  And his duty shall be to lead and save the people " Jomo Kenyatta, the first prime minister of Kenya, is immortalised in Weep Not, Child  The author says, "Jomo had been his (Ngotho's) hope  Ngotho had come to think that it was Jomo who would drive away the white man  To him, Jomo stood for custom and traditions purified by grace of learning and much travel " Njoroge comes to view Jomo as a messiah who will win the struggle against the colonial government 



ground_truth: Technological limitations, resistance to exposing data to scrutiny, and desire to hold onto data for personal use.