The sort function can be used for removing duplicates in DataStage. While running the sort function, the user should specify the option that allows for duplicates and set it to false.  Steps to remove the duplicates are -  Go to the Input page, followed by the Partitioning tab, and select the Partition Type as "Hash".  Select the Id and Date fields as Keys.  Check the "Perform sort" check box under the "Sorting" category.  Under the "Selected" category, set the Usage property for the ID and Date fields. For the Id field, Usage should be "Sorting, Partitioning". For the Date field, the Usage should be "Sorting". Keep the default setting for Options.  Go to the "Stage" page, followed by the "Properties" Tab. Select the Key field as "Id".  Duplicate to Retain as " Last"