There is no sure-shot way to spot a trending filter. What you can do is observe, and if a certain filter is getting repeated, it is trending. You can also go to stories and select any filter → tap the filter name → scroll and select Browse effect gallery → tap various tabs to explore new filters.