Just record instanceof GeoExt.data.LayerRecord && record.get("layer") && record.get("layer") instanceOf OpenLayers.Layer.Vector 

But ... why are you checking it's a geoext layerrecord BTW ?