A man wearing a red vest is walking past a black and green fence.
The man wearing the vest is walking down the street.
You can not infer he is walking down the street.
We have no idea if they are walking down the street, they could be walking anywhere.
You can not infer he is walking down the street.