Q:

What is the expected behavior of _NET_WM_STATE_MODAL? 

I am making a cross-platform windowing layer.  When making window relationship stuffs, I got some trouble on window modality. 
I have read the official spec: Application Window Properties, and some related topics like this: X11 modal dialog.  It seems not sufficient to only set transient-for, but _NET_WM_STATE_MODAL is also required.  So I tried to make small programs that apply this property along with transient-for. 
I firstly made the program that create the window using SDL2, and use X11 stuffs using the fetched native window handle.  But I did not observe any behavior change after the _NET_WM_STATE_MODAL attribute is set: the transient-for target window is still receiving mouse button events, which is not like a modal-blocked parent window that cannot operated by user. 
To avoid potential evil stuffs done by SDL2, I further made the test program using GDK3, which provides ready-to-use wrapper functions.  The behavior is same as the SDL2 program. 
As I did not observed any change before/after _NET_WM_STATE_MODAL is set, what is the expected behavior of that property? 

A:

As I did not observed any change before/after _NET_WM_STATE_MODAL is set, what is the expected behavior of that property? 

That's a question we cannot answer.  It's a hint for the window manager to indicate modality, but, as in most cases, it is up to the window manager to decide what to do with this hint. 
In other words behavior depends entirely upon the window manager and you haven't stated which window manager you were testing with. 
Furthermore, this hint requires the window manager to be EWMH-compliant, which not all of them are or aren't fully.  You can use _NET_SUPPORTED on the root window to see a list of atoms the window manager claims to support.  If _NET_WM_STATE_MODAL isn't listed there, chances are the window manager really doesn't implement this hint at all.  If it is listed, the window manager claims to support it, but a) it might be lying (let's not assume that, though) and b) behavior is up to the window manager.