Fixed a bug in 'rlixlist'

Fixed a nasty bug in 'rlixlist' that lead a segfault in some cases.
Fixed an unnecessary ordering in 'duplicated.data.table'

duplicated.data.table (and therefore unique.data.table) now 10-12x faster than 1.8.10 due to 2 factors: 1) fast order 2) removal of an unnecessary ordering in code.

Updated NEWS. No tests added.
Fix for #5190

Arose while implementing FR #4979. When grouping with .SDcols argument value as a variable, it lead to an error "symbol not subsettable". This is now fixed.

Tests added and NEWS updated.