added duplicate removing method and fixed validation bug