Question: What experiments are proposed to test that upper layers produce context-specific embeddings?
Answer: They measure self-similarity, intra-sentence similarity and maximum explainable variance of the embeddings in the upper layers.
Alternative answer: They plot the average cosine similarity between uniformly random words increases exponentially from layers 8 through 12.  
They plot the average self-similarity of uniformly randomly sampled words in each layer of BERT, ELMo, and GPT-2 and shown that the higher layer produces more context-specific embeddings.
They plot that word representations in a sentence become more context-specific in upper layers, they drift away from one another.