ActiveDisplay: fix possible add>remove>add(nothing happens) condition

Change-Id: I7bb444eced5de0856f60dda7804ee4864546bc9d