The input to vtkDecimate is a vtkPolyData object, and only triangles are treated. If you desire to decimate polygonal meshes, first triangulate the polygons with the vtkTriangleFilter object. The algorithm proceeds as follows. The resulting hole is then patched by re-triangulation.
Iterations proceed until a target reduction is reached or a maximum iteration count is exceeded.
There are a number of additional parameters you can set to control the decimation algorithm. The Error ivar may be increased over each iteration with the ErrorIncrement. Sub iterations are iterations that are performed without changing the decimation criterion. The AspectRatio ivar controls the shape of the triangles that are created, and is the ratio of maximum edge length to minimum edge length. The Degree is the number of triangles using a single vertex.